Method memes, or “Is it complicated enough yet?” …. ***

Everybody knows show-offs. Everybody is also guilty, at least sometimes, of being a show-off. It’s forgivable and often just a childish initial phase for something you newly encountered, and are fascinated by, which will eventually wear off and go away. I religiously follow the games of my 8-year-old’s soccer team. Every kid on that team has watched Lionel Messi and Cristiano Ronaldo on YouTube performing amazing ball artistry, and is busy imitating these giant role models. That’s understandable and good. Occasionally, however, the kids are so mesmerized by the ball artistry that they forget the purpose of winning the game and that pointless tricks do not get them nearer to winning. The coach helps them on their journey and points out this flaw. ‘Wow, you did a rainbow and 3 scissor faints when you had the ball totally unencumbered and could have just run with it… but the other team still played better and won!”

Purposeless artistry in the service of showing yourself off eventually gets old: people will think you are ridiculous and ineffectual. Explaining 1+1 to a child by computing a geometric series is pointless. Trying to knock somebody out by doing some splits and flips first is pointless too (although a Capoeira player might disagree). It gets worse when a whole field of academic inquiry is derailed and performs purposeless artistry as par for the course.

My discipline of neuroimaging analytics arguably is in the grip of purposeless artistry; somebody –I forgot who it was- came up with the fitting term “method meme mania”. It is hard to gauge the scale of the problem. Of course there are vast regions of this field where good science is performed. Also, awareness of the problem is growing too and in a beautiful dynamic of self-correction spawns a good amount of counter didactics. I cannot claim to have researched this comprehensively and only speak of my own anecdotal impressions, attending conferences and reviewing papers for academic journals. I am also too timid to name individual techniques or papers lest it gets me in trouble professionally. Also, it would be unfair to single out individual papers, since it’s a more general malaise. So there is all that, granted.

As a reviewer I am currently most struck by Graph-Theory applications. Graph Theory is a wonderful field of mathematics going back to the 18th century, with many applications in diverse fields where network phenomena appear. Neuroimaging analytics has benefited from the contributions of several researchers who have further developed, popularized and successfully applied Graph Theory including, but not limited to, Olaf Sporns, Ed Bullmore, Dani Bassett, Alex Fornito, and several others.

I am not singling out Graph Theory, since there are other techniques de jour that would equally qualify here. Graph Theory happens to be the most visible example in my personal universe. As a reviewer for neuroimaging journals, I have maybe reviewed a dozen or so Graph-Theory papers, by which I mean these papers featured applications of Graph-Theory measures to neuroimaging data with the hope of explaining something meaningful; they usually did not do any theoretical development work for Graph Theory.

When using and advocating a new technique to my mind there are two suitable justifications: (1) the technique addresses a problem better than other techniques out there, and lends a mechanistic insight that was hitherto impossible to address, (2) the technique is agnostic to scientific questions of mechanisms, but it can produce results from which superior biomarkers can be constructed that work better for diagnosis or prognosis than hitherto existing biomarkers. Detailed knowledge of mechanisms, while nice, is not a prerequisite for clinical utility. So (2) would still be a worthy goal, even if (1) is out of reach. The problem with my sampling of Graph-Theory papers: they often meet neither bar (1), nor bar (2). Rather, they seem to be motivated by the rationale “Let’s do something cool with Graph Theory.” Neither are they trying to gain a mechanistic insight, nor are they trying to construct a superior biomarker for something, which would necessitate a comparative survey in which the newly derived marker is pitted against pre-existing choices in strict comparisons of predictive utility. While such comparative evaluation is not very hard to do, it is rarely undertaken. Very often the technique is motivated sui generis with the main rationale that other researchers have used similar techniques. Maybe the current authors added a particular complication for intellectual ownership, but the main justification is that somebody else used a similar technique.

For Graph-Theory applications, often the authors compute one particular measure, like local or global efficiency, centrality (="hubness") measures, or something more complicated, and show a correlation with a clinically meaningful variable. The paper concludes with a statement on the diagnostic relevance of this measure and possible further research where the utility is mapped out in more detail. Comparative out-of-sample performance pitted against other less sexy measures, like the mean correlation strength of all possible pairs within the graph, is usually not undertaken. A mechanistic speculation about the relevance in mechanistic terms is sometimes supplied, but it often it is a just-so story given after the fact.

As I mentioned, these considerations do not just apply to Graph Theory. Many other approaches qualify for my criticism here: neuroimaging analytics operates in a high-dimensional data world eager for methodological novelty. Ever more derivative approaches using higher-order moments far removed from the raw data, with doubtful statistical robustness, never mind any mechanistic justification, abound. A good amount of intellectual effort in peer review and primary authorship is thus spent on such “liberal arts with computers”. There might be nothing wrong with this endeavor and it might produce some useful insights, but it cannot strictly be labeled “science” any longer. It might also not be exactly what the public has in mind when it dispenses research dollars to the NIH budget.