Double dipping, over-fitting or hindsight bias – different faces of the same stochastic illiteracy

Many scientific disciplines from the life sciences, psychology, ecology, etc. face a “replication crisis”. The reasons for this crisis are many and varied; one big one that seems obvious to yours truly is a publication system that encourages spectacular results from under-powered studies which might have engaged in “data dredging”, HARKing (=hypothesizing after results known), or double dipping. In short: good scientific practice was compromised for career success. In a tiny minority of cases this happens due to outright fraught, but most often it is just little nudges that push the researchers down the path of least resistance. The topic has been discussed at length by authors like Ed Vul, Niko Kriegeskorte, Andrew Gelman and others. The books of N. Nassim Taleb are essentially one long irreverent meditation on the problem, particular in the finance sector where people were beholden to an empty formalism of Gaussian models with catastrophic consequences in the 2008 mortgage crisis.

I will not rehash this material in detail since excellent papers and tutorials exist to avoid double dipping in practice. What is important here is that these considerations can be appreciated by everybody, not just by statistically inclined readers. Conceptually, double dipping or over-fitting is quite simple. I will highlight a few thought examples that represent extreme versions of double dipping. Real-life encounters are usually not that extreme and clear-cut, but understanding the extremes will sharpen your senses for lesser violations.

Double dipping means doing something to the data that invalidates the subsequent statistical test, usually without awareness. The data might be transformed or filtered in certain ways that invalidate crucial assumptions. Further, it also includes applying overly complex models or algorithms to the data such that the explanatory power of the obtained results in new data is vanishing. Hence “replication crisis”.

As I said, this post is an attempt at a non-quantitative explanation with some outrageous examples that hopefully make double dipping more understandable to the non-technical reader. Some of these examples, particularly the most illustrative ones, have been stolen from others.

Let us look at a simple example. It is so silly as to be totally obvious. Imagine that for a country-wise comparison of male height between the Netherlands and South Korea, a sample of 1,000 males are randomly picked from both countries. A simple T-test could now be run to come up with p-level for the rejection of height equality between the countries. I would venture that the Netherlands might win out, since the average Dutch male height is 6 feet 1. Imagine now that the researcher gathering all data was partial to South Korea. Instead of randomly sampling 1,000 males, the researcher instructed his team to do a lot more work and sample 20,000 males in South Korea, and of those only pick the 1,000 tallest. A simple T-test now shows that South Korea is on average slightly taller than the Netherlands, p=0.02. Obviously, this is fraudulent. The T-test assumes a random sampling plan: we cannot engage in some prior data selection that clearly messes with the characteristic (=height) which will subsequently be probed with a statistical test. This is an extreme case of double dipping: the data were manipulated, in this case with an intent for a particular outcome.

In real-world cases of double dipping the intent is usually lacking, and data are probably not blatantly pre-treated in such a fashion. Instead the error creeps in at the level of the statistical inference itself. Most often, it comes in the form of an iterative procedure that unfolds with ad-hoc decisions by the analyst that were not carefully mapped out beforehand. Such an algorithmic recipe though is crucial to estimate false positives: you need a fully specified plan of the whole analytic chain to see how likely it is that you find results as good as yours even in random junk data. If you dish out different analyses on the fly and see “where the data leads you” this might qualify as art, but a rigorous inferential assessment is not really possible. Often, in the absence of a clear prior analytic plan, people might be tempted to follow a trial-and-error approach, until they converge on something “that works”, i.e. yields a statistically significant result. The problem is that the nominal p-level from that last statistical test will be invalid since you did not adequately take into consideration all the stuff you did prior to this statistical test.

We can now turn to a few other really outrageous examples. This is one of my favorites, taken from a paper by E. Vul and N. Kanwisher. Imagine your friend rolls a die 4 times to obtain 3,2,5,6, in that order. He cries “Wow, the probability of this combination of throws in that order is only (1/6) ^4, i.e. less than 0.1 %.” He keeps repeating this many times, every time yelping with astonishment about the vanishing probability of getting this particular throw combination. This feels silly, and it is. But it might be a little hard to pinpoint exactly why this is so silly: after all, the formula for the computation of probabilities is correct, no?

The details of the formula are a red herring: the problem is that no formula applies here at all. Why not? Because computation of any probability only makes sense in relation to a prior announcement of the throw combination. To make this a little more tangible: think about how often somebody with a non-winning lottery ticket would show their astonishment at the improbability of the winning number combination?

The reader can appreciate that the crucially missing prior prediction underlies a lot of “Monday morning quarterback” media discussions suffering from clear hindsight bias, leading to apparent reversals of causality: “The terrorists attempted to smuggle explosives through security in their underwear.” “Oh, but why did the TSA not screen people’s underwear?” - "Tech Stock X crashed right before I wanted to cash out and sell!" "Why did you not sell a little earlier?"

Double dipping in data analysis is similar, though less extreme. It’s not that no prior analysis plan and hypothesis exist, it’s just that the analytic recipe is not pinned down sufficiently and there are too many shifting parts filled in ad-hoc, possibly guided by the analyst’s desire to see a particular publication-worthy result. A. Gelman coined the fitting phrase “researcher’s degrees of freedom”. This is similar to a poorly written and flabby movie script: plot lines, superpowers, etc. are made up on the fly without prior buildup, resulting in a very unsatisfactory viewing experience.

As many of the above mentioned authors make clear: this is not a technical issue only appreciated by quants, this is elementary reasoning. One cannot select evidence to fit a particular hypothesis and then expect to perform a genuinely unbiased test of the same hypothesis in the selected data. This is intuitive for most people: for an unbiased review of a book, for example, you should consult Kirkus Reviews rather than the promotional blurbs on the book jacket.

Sharpening your probabilistic understanding does not mean that you have to become fluent in math formulae. N. Nassim Taleb has done the most to point this out. Meeting a portfolio manager with a fantastic track record over every one of the last 5 years is impressive. It’s much less impressive if you learn that even one bad year would have led to the manager’s firing and that he is only one of many other managers at this giant investment firm. As Taleb explains, even with chance performance somebody will get things right 5 years in a row if there are enough starting competitors. If having a bad year leads to a manager’s elimination, then by necessity all surviving competitors will have perfect track records. This bias alone can explain the track record, and no great skill is necessary.

We can repeat this point in different guises. If a clairvoyant predicts this week’s lottery numbers correctly, I am indeed quite impressed. If I subsequently learn that the clairvoyant did not just make one prediction, but instead generated all possible 6! predictions, and that I, by some freak chance, witnessed only the correct prediction, I am much less impressed.

Taleb pushes this to wider contexts. A lot of decisions cannot be represented just as picking one option of a computable number of alternatives: real life cannot be framed as such a combinatorial calculus. However, vastly over-estimated agency + survivor bias (and the penchant and rewards for extreme rhetorical confidence) might explain a lot of successes in US corporate culture. This is hard to prove since there are no rigorous experiments of course, but from friends’ anecdotes this rings somewhat true for me. Getting a lucky break is very easy to dress up as personal hard work and skill post-hoc, particularly if the cultural Zeitgeist does not really believe in luck.

So double dipping and seeing patterns in random noise might be a lot more endemic than everybody realizes. Taleb might have a point here. For science though, some simple safeguards are easy to adopt. By now they are quite commonplace, thanks to all the authors credited in this post:

1) Specify your goals and analysis plan beforehand, ideally even before data collection. This enables easy type-I error computations in Monte-Carlo simulations or permutation tests,

2) Try to predict things out of sample, and also think beforehand what exactly constitutes a successful prediction. (This exercise already is very insightful.)

3) Don’t be apologetic about data exploration, but mark it clearly: most problems of non-replication come from dressing up exploration as confirmation for the appearance of inferential rigor.

Awareness of double dipping in science, and incontrollable randomness in every-day life hopefully grows, and society becomes more scientifically and stochastically literate. This would be a very positive development. For every-day life, the ever increasing choice of options with the illusion that you are in total control for optimizing your choice, very often leads to regret and second-guessing of decision making, in the worst cases turning into feelings of failure. You might heave a sigh of relief upon learning that you are not as in control as you think. Take some advice from the late Mose Allison: “I don’t worry about a thing, because nothing’s gonna be alright!”.