New York Times on the Probability of Rare Events

The New York Times has published a short article about the commonness of rare events.  The article discusses the fraudulent scheme of a psychic sending out emails predicting the outcome of a ball game.  After 8 successful ballgame predictions, the psychic asks for $10 before providing you the next prediction for the ninth game.  The article states that the psychic is guessing, so how did he do it? 

This is a version  of a well known scheme that works like this.  The psychic sends out batches of 256 emails, 50% of them predicting the first team will win, 50% predicting the second will win for each ball game.  Half of the recipients receive a correct answer and half receive the wrong answer.  The next week, the psychic sends an email only to those that received the correct answer in the first week.  Again half of these are told the first team will win, half are told the second team will win.  This continues for eight weeks.  At the end of the eight weeks, one recipient will have received 8 correct predictions and a request for $10 for the next prediction.If the psychic starts with 25,600, emails, after 8 ballgames, there will be 100 recipients who have received perfect predictions.

The psychic’s success rate can be dramatically improved if he weights his predictions to the favorites for each game.  For example, suppose the average game has one team with betting odds of winning of 1.5:1, and that these odds are a reasonable predictor of the outcome.  This means that the psychic only has to send out 59.5 emails (instead of 256) on average to produce one recipient with 8 good predictions.  This is calculated as 59.5*(1.5/(1.5+1))^8 = 1.  If the psychic is choosing only one game from a weekly roster of games, he could choose even more lob-sided games and further improve the chance of producing a winning series of predictions.

A version of this occurs naturally in the financial markets, although with less of an intent to deceive.  Assume that there are 1,024 traders.  If each makes a key investment decision every 6 months, over a four year period, there will be roughly four that have a perfect record, assuming that each correct decision occurs with a probability of 50%.  These four traders are likely to be considered Wall Street geniuses.

Now real life is always more complicated than one might expect.  With traders, there is a strong likelihood that at least some of them are smarter than the rest and are more likely to make good decisions. Let’s assume that “Good” traders are 60% correct in their decisions, “Average” traders are 50% correct, and “Poor” traders are 40% correct.  Let’s further assume that the split of Good, Average and Poor traders is 10%, 80%, 10%.  The question that is most interesting is: if you come across a trader with 8 correct decisions, what is the probability that that trader comes from the “Good” category.  This is the key decision an investor must make when  looking to place his or her funds with an investment manager or trader (or a manager must make before paying out fat trader bonuses).

This can be solved in the following way:

Trader Strength	Probability of correct decision	Proportion of traders	Probability of 8 correct decisions	Percentage of decisions which are correct	Probability Good Decision came from the selected Class
Column	a	b	c	d	e
Good	60%	10%	1.7%	0.2%	34.5%
Average	50%	80%	0.4%	0.3%	64.2%
Poor	40%	10%	0.1%	0.0%	1.3%
Total		100%		0.5%	100.0%

Column c = a^8

Column d =b * c

Column e = d / (Total column d)

The model suggests that if you come across a trader with an astounding run of 8 correct decisions, you only have a 34.5% chance that that trader is “Good” and not “Average” or “Poor”.

Success and Failure in Medical Testing

There is nothing quite as unsettling or at times scary as getting back a medical test result that suggests that you have a particular disease or one of your bodily parts is not functioning correctly. Your despair is undoubtedly deepened when your doctor inevitably tells you that the test that has uncovered your sad predicament is in fact very accurate. “97% accurate,” the doctor might say.

Such bad news is faced by people everyday. Yet a little understanding of probability can restore some hope to the situation. This is particularly the case for tests that are conducted for unusual diseases or for which you have no history of potential exposure. So how do you figure out what your real risk is?

The first thing to notice is that the doctor is quoting the accuracy of turning up a false positive, namely that the test indicates that you have the disease when in fact you don’t. What is not being quoted is the conditional probability of given that you have a positive test result, what is the probability that you have the disease. This might sound like the same thing, but in fact, the two probabilities are very different.

To make further headway into the issue, it is important to know what the probability of occurrence of the stated disease is. Let’s take as an example testing for cancer. The following example uses made up assumptions to demonstrate the point, and would not represent actual risks.

Say the cancer test says that you do not have cancer 97% of the time, when in fact you do not, and says that you do have cancer 99% of the time, when in fact you do have cancer. The first measure (97%) is called the “specificity” of the test. The second measure (99%), is called the sensitivity of the test.

Now assume that two in a hundred people of the test taking age do in fact have cancer. Armed with this information, we can now find the solution with the help of a magic probability box. The box looks at the intersection of having and not having cancer, with the results of the test.

We start by cross multiplying the proportion of the population having and not having cancer with the sensitivity and specificity of the test, respectively.

	Have Cancer	No Cancer	Total
Test says “No Cancer”	2% x (100%-99%)	98% x (97%)
Test says “Cancer”	2% x (99%)	98% x (100%-97%)
Total

Next we calculate the results and add up the rows and columns.

	Have Cancer	No Cancer	Total
Test says “No Cancer”	0.02%	95.06%	95.08%
Test says “Cancer”	1.98%	2.94%	4.92%
Total	2.00%	98%	100.00%

Note that the bottom row shows the proportion of the population having and not having cancer and that the total in the two calculated columns and rows both add up to 100%. That is, the four calculated cells add up to 100% and these four cells represent the entire universe of “event” outcomes.

Finally, we are ready to answer the question: If the test says cancer, do you in fact have cancer. The probability of this is

= Prob(having cancer and the test says you have cancer) / Prob(test says you have cancer)

= 1.98% / 4.92%

= 40.24%

Now what is the probability that if the test says cancer, you do not have cancer. The probability of this is

= Prob(not having cancer and the test says you have cancer) / Prob(test says you have cancer

= 2.94% / 4.92%

= 59.75%

We are left with a remarkably counter-intuitive result: You are more likely not to have cancer, than to have cancer despite the so called “accuracy” (sensitivity and specificity) of the test appearing to be quite high, namely 99% and 97%. Why does this happen? It occurs because the low incidence of the disease in the population is in fact lower than the error rate of the test. The test is more likely to be erratic than for you to have cancer.

The implications of this simple example are profound in how we conduct our lives. There is of course the unnecessary worry and concern that results of medical tests produce. More problematic is if the conduct of the test has a cost. This is particularly the case where the cost is not just monetary but physical.

Take for instance the test for Down Syndrome with the use of an Amniocentesis. An amniocentesis collects amniotic fluid, the fluid in the womb, and tests the fetal cells for Down Syndrome. The test is done in the 16-18th week of pregnancy. The test increases the risk of miscarriage by about 0.75%. The test is often recommended for pregnant women 35 years and older. The risk of Down increases with age and is about 1/250 at the time of the test for a 35 year old and 1/20 for a 45 year old.

Let’s assume we have 250 woman of the same age and risk class taking the Amniocentesis and we want to find out the probability that the test will be successful. The question is how to define success. This might be as much a qualitative as a quantitative issue, but one way to do it might be:

• Success = discover Down, Down exists
• Failure = Do not have Down but have a miscarriage as a result of the test
• Neutral = Other outcomes are deemed to be “neutral” and we take the opinion that these are neither good nor bad. This category relates to pregnant woman who do not have a Down pregnancy and do not have a miscarriage from the test.

To simplify matters, we assume that the sensitivity and specificity of the test are both 100%. Now we create a new magic probability box and look at the risks for a 35 year old, making the following assumptions:

• probability of a miscarriage from test for all pregnancies: 0.75%
• probability of Down in population of 35 year-old pregnant woman at time of test: 1/250

	Miscarriage from Test	No Miscarriage from Test	Total
Down	0.75% (1/250)	99.25% (1/250)	1/250
No Down	0.75% (249/250)	99.25% (249/250)	249/250
Total

Completing the table gives:

	Miscarriage from Test	No Miscarriage from Test	Total
Down	0.00003	0.00397	0.004
No Down	0.00747	0.98853	0.996
Total	0.0075	0.9925	1.000

For a group of 250 women, we have

• Successes = 250*0.004 =1
• Failures = 250*0.00747 = 1.8675 = miscarriage for women without Down fetus
• Total successes and failures = 1 + 1.8675 = 2.8675

We note that there are more unnecessary miscarriages than Down discoveries!

We now seek the probability of “success” of the test, given that the test produces either a success or failure, according to our prior description.

• Prob (success, given a success or failure in the test)
  • = 1 / (1+1.8675)
  • = 34.9%
• Prob (Failure, given a success or failure in the test)
  • = 1.8675 / (1+1.8675)
  • = 65.1%

The result is quite dramatic if you consider that failures result in an unnecessary miscarriage, and leads to the question whether the test produces more harm than good. To answer that question, probability calculations alone will not suffice.

For a 45 year old, the success probability increases to 87.5%.

The advice most doctors give their patients on deciding whether to do the test is to quote the extra risk of miscarriage of 0.75%. This seems very small. However, when you translate this small extra risk to the conditional probability of success or failure, the matter becomes far from clear. The issues become even more complex when you make a value judgment of whether the benefit of a discovery of a Down pregnancy is a better or worse outcome than the causation of miscarriage.

DISCLAIMER: The above analysis does not recommend you should not get tested for cancer (where I made up the probabilities above), or not have an amniocentesis. The above is just a simplified model of how you can think about these matters, and could be quite inappropriate for your circumstances. Individuals should discuss these matters with their doctors who might be aware of many other factors. You should not rely on the simplified models only when making life altering decisions. Speak to your doctor!!

Random Variables, Probabilities and the Power of the Unknown

I have started this blog as a recreational investigation into the world of random variables, randomness and probabilities. I will be exploring interesting ways in which probability and randomness affect our lives and will be looking at ways of solving easy, difficult and unusual probability questions. The blog will explore ways of predicting the future, and shedding light on the unknown. Probability techniques will be discussed as a way of improving critical thinking and uncovering flaws in our day-to-day decision making processes.

I hope this blog will be the product of a collaborative process between me and the blog's readers. As such, I encourage all readers to contribute ideas, post questions and challenges, offer new ideas and solutions.

Feel free to email me at x@randomvariable.us with any thoughts or questions.