molecular biology and kuhn . . .
Microparadigms: chains of collective reasoning in publications about molecular interactions. A Rzhetsky, I Iossifov, JM Loh, KP White. (bio and gen people from columbia and yale) PNAS 2006
i respect modeling more and more after taking the math modeleing class in my last semester (even though my own model was a big unsuccess [push that thing toward submission again, or muck it?])
the authors analyzed millions of published statements about molecular interactions (apparently all these things are stored in a searchable database of papers, and the extraction process was pretty much automated). then they constructed strings of related statements over time. simple logical relationships between molecules (protein A activates protein B) in these statements were tagged with 1's and 0's (for true/false), and then the dependence of future links on prior links can be modeled.
"We found that published statements, regardless of their verity, tend to interfere with interpretation of the subsequent experiments and, therefore, can act as scientific "microparadigms," similar to dominant scientific theories [Kuhn, T. S. (1996)..."
the authors dub the positive influence of existing findings on subsequent findings "experimental momentum." how this works: prior results frame the conceptions of a new 'space' of molecular interactions, so that alternative interpretations are irrationally deweighted priors. reviewers only will accept findings framed in prior findings, making repetition more likely. positive results are much more likely to be published than negative results (e.g. we found that my eyebrows do not interact with molecule x) - though the authors take this into account in their model.
they put forth several model scenarios to demonstrate the effectiveness of their model, and then move on to the results when using real data. dozens of 'experiments' are initialized at timepoint 1 and these are then more-or-less repeated in subsequent steps. they basically can model something like the coupling between prior and current work by measuring how much subsequent expertimental statements are influenced by prior statements, and how much this influence deviates from chance (chance of many repeated same results, etc.) e.g. if error is 5% on any experiment, the chances of 20 similar experiments reproducing the same result is .05^20 - rather unlikely, unless the "momentum" of prior results is high.
-one, where each experiment was conducted in complete isolation, not influenced at all by any others. results: each experiment is predictably as accurate as a normal p<.05 error rate.
-two, where each experiment is stronly influenced by prior results. strong conformity. resulting long strings of replications and infrequent opposing findings - which are not themselves replicated.
-three: super anti-conformism, where each finding has a net negative effect on replications. "believe no one (but yourself)."
-four, where each experiment is strongly influenced by prior results, but when a contrary finding is made, everyone jumps onto that bandwagon immedialy. ("anticonformism with an inferiority complex")
-five: the ideal scientific universe, "mild scepticism", where subsequent results are positively influenced by prior results (experimental momentum is greater than zero) but this influence is weak. experimenters trust their data much more than others, but try to align it with existing work. in this version, the model allows the authors to pick the optimal variables that ensure the fastest convergence on the correct result.
findings from modeling the millions and millions of interactions: an optimist's universe and a pessimists universe:
An evaluation of the optimum parameters under our model (see Model Box) indicated that the momentums of published statements estimated from real data are too high to maximize the probability of reaching the correct result at the end of a chain. This finding suggests that the scientific process may not maximize the overall probability that the result published at the end of a chain of reasoning will be correct. ... our computations indicate that our data set can be interpreted in two very different ways (two "alternative universes"): one is an "optimists’ universe" with a very low incidence of false results (<5%),>90%). Our computations deem highly unlikely any milder intermediate explanation between these two extremes.
...
If the problem of convergence to a false "accepted" scientific result is indeed frequent, it might be important to focus on alleviating it through restructuring the publication process or introducing a means of independent benchmarking of published results."
? i want more discussion of their conclusions. the paper ends very briefly with talk of their models. maybe more discussion of the meat in online supplements
? this hits on something that has always interested me in the philosophy of science, strongly influenced by Kuhn and Paul Churchland (and maybe stu kaufman?): how ideas evolve in societies. Churchland, translated into this realm (most of his stuff is about individual minds/neural networks) is all about random walks through a space of theories. random in a sense, but influced by force of natural selection - weak survival biases dependent upon chance, a bully pulpit, etc, etc. hell, i even hacked together an Anthro model about evolutionary dynamics in a population of minds analogous to evolutionary and population dynamics in ecosystems on connected but separated “archipelagos of minds.” but this is, to my knowledge, the first attempt at a quantification.
? applications to other sciences. i.e. mine. well, we don't really replicate stuff. too f'ing expensive to redo someone else's imaging study, and we don't get published again if we do anyway. so i would conservatively estimate the error rate in the neuroimaging field to be 20%.
note to self: after moving, dig up old feyerabend.
note to others: check out the guy's crazed eyebrows on the backjacket of, say, 'the end of reason'.
i respect modeling more and more after taking the math modeleing class in my last semester (even though my own model was a big unsuccess [push that thing toward submission again, or muck it?])
the authors analyzed millions of published statements about molecular interactions (apparently all these things are stored in a searchable database of papers, and the extraction process was pretty much automated). then they constructed strings of related statements over time. simple logical relationships between molecules (protein A activates protein B) in these statements were tagged with 1's and 0's (for true/false), and then the dependence of future links on prior links can be modeled.
"We found that published statements, regardless of their verity, tend to interfere with interpretation of the subsequent experiments and, therefore, can act as scientific "microparadigms," similar to dominant scientific theories [Kuhn, T. S. (1996)..."
the authors dub the positive influence of existing findings on subsequent findings "experimental momentum." how this works: prior results frame the conceptions of a new 'space' of molecular interactions, so that alternative interpretations are irrationally deweighted priors. reviewers only will accept findings framed in prior findings, making repetition more likely. positive results are much more likely to be published than negative results (e.g. we found that my eyebrows do not interact with molecule x) - though the authors take this into account in their model.
they put forth several model scenarios to demonstrate the effectiveness of their model, and then move on to the results when using real data. dozens of 'experiments' are initialized at timepoint 1 and these are then more-or-less repeated in subsequent steps. they basically can model something like the coupling between prior and current work by measuring how much subsequent expertimental statements are influenced by prior statements, and how much this influence deviates from chance (chance of many repeated same results, etc.) e.g. if error is 5% on any experiment, the chances of 20 similar experiments reproducing the same result is .05^20 - rather unlikely, unless the "momentum" of prior results is high.
-one, where each experiment was conducted in complete isolation, not influenced at all by any others. results: each experiment is predictably as accurate as a normal p<.05 error rate.
-two, where each experiment is stronly influenced by prior results. strong conformity. resulting long strings of replications and infrequent opposing findings - which are not themselves replicated.
-three: super anti-conformism, where each finding has a net negative effect on replications. "believe no one (but yourself)."
-four, where each experiment is strongly influenced by prior results, but when a contrary finding is made, everyone jumps onto that bandwagon immedialy. ("anticonformism with an inferiority complex")
-five: the ideal scientific universe, "mild scepticism", where subsequent results are positively influenced by prior results (experimental momentum is greater than zero) but this influence is weak. experimenters trust their data much more than others, but try to align it with existing work. in this version, the model allows the authors to pick the optimal variables that ensure the fastest convergence on the correct result.
findings from modeling the millions and millions of interactions: an optimist's universe and a pessimists universe:
An evaluation of the optimum parameters under our model (see Model Box) indicated that the momentums of published statements estimated from real data are too high to maximize the probability of reaching the correct result at the end of a chain. This finding suggests that the scientific process may not maximize the overall probability that the result published at the end of a chain of reasoning will be correct. ... our computations indicate that our data set can be interpreted in two very different ways (two "alternative universes"): one is an "optimists’ universe" with a very low incidence of false results (<5%),>90%). Our computations deem highly unlikely any milder intermediate explanation between these two extremes.
...
If the problem of convergence to a false "accepted" scientific result is indeed frequent, it might be important to focus on alleviating it through restructuring the publication process or introducing a means of independent benchmarking of published results."
? i want more discussion of their conclusions. the paper ends very briefly with talk of their models. maybe more discussion of the meat in online supplements
? this hits on something that has always interested me in the philosophy of science, strongly influenced by Kuhn and Paul Churchland (and maybe stu kaufman?): how ideas evolve in societies. Churchland, translated into this realm (most of his stuff is about individual minds/neural networks) is all about random walks through a space of theories. random in a sense, but influced by force of natural selection - weak survival biases dependent upon chance, a bully pulpit, etc, etc. hell, i even hacked together an Anthro model about evolutionary dynamics in a population of minds analogous to evolutionary and population dynamics in ecosystems on connected but separated “archipelagos of minds.” but this is, to my knowledge, the first attempt at a quantification.
? applications to other sciences. i.e. mine. well, we don't really replicate stuff. too f'ing expensive to redo someone else's imaging study, and we don't get published again if we do anyway. so i would conservatively estimate the error rate in the neuroimaging field to be 20%.
note to self: after moving, dig up old feyerabend.
note to others: check out the guy's crazed eyebrows on the backjacket of, say, 'the end of reason'.
0 Comments:
Post a Comment
<< Home