Base-rates, or when applying almost precise medical tests to healthy people goes awry

This example has been publicized most effectively by Gerd Gigerenzer in his book “Risk savvy”. The book is worth reading because it shows the bad confluence of general risk illiteracy with too much trust in the power of high-tech diagnostics which can lull people into too much certainty with beautiful graphics etc. Gigerenzer suggests some very practical steps to take into account Bayes’ theorem and draw simple decision trees. He also advocates the positive predictive value (PPV) as a more meaningful metric for diagnostic accuracy of a medical test than sensitivity and specificity. This is worth going through in detail since most physicians are profoundly confused about it. I don’t blame them in the least; it is easy to get confused.

Let’s recall that the only quantity that can be given as an absolute about a medical test is the sensitivity or specificity. These two mean the following:

Sensitivity = probability that a sick person is accurately diagnosed as sick

➔ P(D=1|X=1)

Specificity = probability that a healthy person is accurately diagnosed as healthy

➔ P(D=0|X=0)

Here we denoted the diagnostic verdict as D=0 or 1, and the true disease state as X=0 or 1. One can appreciate that both sensitivity and specificity are not the most helpful metrics for you as patient. Why? Because they tell you about the test more than about you. What you would like to know is the positive predictive value, i.e. the probability that you are really sick having received a positive diagnosis: Positive predictive value = probability that a person with a positive test result is actually sick ➔P(X=1|D=1).

This is really what you and your doctor would like to know. The problem is that it depends on the base rate, or prior, of the disease in people who are just like you, i.e. have your age and demographics. We can resort to Bayes’ Theorem to make this clear and write down the joint probability:

P (X=1, D=1) = P(X=1|D=1) * P(D=1) = P(D=1|X=1) * P(X=1)

Here we just have written the joint probability density as two different products with different prior and posterior probabilities. To isolate the PPV, we can divide the joint probability by the prior for a positive diagnostic result:

PPV= P(X=1|D=1) = P(D=1|X=1) * P(X=1) / P(D=1).

We can plug in some numbers now; assume a sensitivity and specificity value of 95%, i.e.

P(D=1|X=1) =95%

P(D=0|X=0) =95%,

and let’s further assume the person undergoing the test comes from a low risk group, i.e. P(X=1) =1%. We can stop here and ask ourselves: should the base rate for the disease matter, i.e. is the positive predictive value dependent on the person the test is applied to? Unless the test is perfect one would assume: YES. A positive test result for an otherwise healthy 20-year old would probably cause you much more skepticism than for a 90-year old.

But in order to verify this intuition, we also need the prior probability for the biomarker P(D=1). Computation of this prior is somewhat involved and means you have to do a little work and cannot easily arrive at the answer just from the test accuracy alone. I would guess that this difficulty is the major reason medical test results are hard to understand by patients and physicians alike. I encourage you to check the following BBC article about Gerd Gigerenzer which has a wonderfully intuitive example with decision trees: http://www.bbc.com/news/magazine-28166019.

P(D=1) depends on both sensitivity and the complement of specificity, also termed “false positive rate” (=P(D=1|X=0) =5% for our biomarker). The formula is the following:

P(D=1) =P(D=1|X=1) * P(X=1) + P(D=1|X=0) * P(X=0)

In other words, positive test results are obtained for correctly diagnosed people as well as for healthy people who are misdiagnosed. 99% of all people are healthy. We can plug in the numbers:

P(D=1) = 95% * 1% + 5% * 99% = 5.9 %

Inserting everything now into our erstwhile formula for PPV gives:

PPV= P(X=1|D=1) = P(D=1|X=1) * P(X=1) / P(D=1) = 95% * 1% / 5.9% = 16.1 %

Yep, 16.1% (!) is the sobering answer. No medical test is perfect, and even great values for specificity and sensitivity will still result in a low PPV if the test is applied to generally healthy people. Some diagnostic tests will lead to great anxiety, early intervention and biopsies, i.e. there might be really negative consequences, only because you wanted to be diligent and follow your doctor’s advice and test early and often even in the absence of symptoms.

In the BBC article Gigerenzer points to a related fallacy for cross-country comparisons of medical efficacy. Often the survival rate, i.e. the fraction of patients alive 5 years after initial diagnosis, is the most important metric that is tracked and compared between countries. This is an unhelpful and bogus comparison. Why? Because the example above shows that the positive predictive value is conflated with how liberally the test is employed in the general population. Comparing different countries with different test and treatment guidelines thus compares apples to oranges. Testing people without any medical indication will result in a very low PPV. If the test result then leads to treatment, you will treat a lot of healthy people. For an extreme example, imagine you declare everybody as suffering from prostate cancer - well maybe only the male half of the population. Thus every male would get treated. Your survival rate for prostate cancer would look absolutely fantastic, close to 100%, while your disease-specific mortality might be very similar to a country that employs the test only sparingly for people with symptoms. That latter country might not achieve better treatment outcomes than the outrageously treat-happy country for people who are actually sick, but the disease-specific mortality between the two country might be identical.

As the BBC article somewhat relishes, Rudolf Giuliani committed this error when, upon learning his own prostate cancer diagnosis, he vocally extoled the US medical system since the prostate-cancer survival rate appeared more than double that of the British National Health Service. However, when tallying disease-specific mortality, i.e. the per-capita death rate from prostate cancer, the US and the UK were identical! This is not surprising: liberally treating healthy people in the end does nothing for the people who are truly sick.