Features 21stC
home page

E T H I C S I N R E S E A R C H

What's the verdict
on peer review?

The judgment of colleagues plays a critical part
in how grants are distributed, journal articles are
selected, and careers are formed. Yet this system may
raise ethical dilemmas. 21stC explores how peer review
really works, and how it might work better

By TOM ABATE

A YOUNG CARDIOLOGIC researcher published his findings in the world's most prestigious medical journal-which later retracted the paper when auditors found that he had cooked his data. An investigator studying drug treatment for hyperactive children did likewise. A top American immunologist was chastened for failing to adequately credit overseas colleagues in the identification of a new virus. And one of America's foremost scientists, at first supported by deferential peers against a largely defenseless junior colleague in a dispute over the publication of questionable data, was eventually forced to resign his prestigious post when the data were determined to be bogus. Why didn't the peer review system prevent these instances of unethical behavior? Was the system, as many have argued, part of the problem?

The ethical foundations of the peer review process have come under heated questioning in recent years. Women and minorities have charged that an old boys' network denies them fair due. Smaller or relatively obscure institutions feel slighted by larger rivals. And young scientists have complained that the system stacks the deck against their work, even though their whole careers are at stake. (Particularly troublesome for the younger investigator is the Oz-behind-the-curtain effect: Reviewers often work anonymously, giving them greater opportunity to act arbitrarily. The reviewee usually has no comparable curtain to stand behind.) Even Congress has investigated peer review, spurred by the well-publicized frauds and coverups of the 1980s, which the ostensibly self-policing mechanisms known as peer review had failed to disclose or prevent.

From 1985 to 1994, NIH's success rate dropped from a third to a quarter. "We're talking about the pain threshold"

Strictly speaking, peer review describes various processes by which panels of experts read and grade grant applications, issuing opinions that influence funding decisions. Reviewing articles for journal publication is more accurately called refereeing, but the term "peer review" describes this editorial process as well. All these activities pose ethical challenges. Whether grading grants or screening articles, a peer reviewer must preserve scholarly integrity by rising above the three deadly sins of intellectual life: envy, favoritism, and the temptation to plagiarize. For, as science historian Horace Freeland Judson observed last year in the Journal of the American Medical Association (JAMA): . . .the persons most qualified to judge the worth of a scientist's grant proposal or the merit of a submitted research paper are precisely those who are the scientist's closest competitors." (1)

PAST AND PRESENT PRACTICES

HOW DID PEER REVIEW, with all its built-in conflicts, become the judicial system of the intellectual world? Michael Crowe, a science historian at the University of Notre Dame, says that by the 1840s, the Royal Society in London was occasionally asking scientists to read papers submitted for publication in its Philosophical Transactions. Sally Gregory Kohlstedt, historian of science at the University of Minnesota, says editors of early American journals had the same informal attitude toward peer review, using it only when they lacked the expertise to judge submissions. Peer review took a tentative step into the realm of funding after 1863, when the newly formed National Academy of Sciences began receiving private gifts to support research. It formed ad hoc committees of experts to distribute the awards. But Kohlstedt said not until World War I, when the federal government began supporting scientists through the National Research Council and questions arose about fair distribution of a public resource, did these private committees start to bear the burden that peer review carries today. "I don't think ethics played much of a role until the 20th century, when more money became available from private and public pockets," Kohlstedt says.

Thus, by the post-World War II science boom, peer review had become accepted practice. "It came into full force after the war with the establishments of the National Science Foundation and the National Institutes of Health," says Jonathan R. Cole, provost of Columbia and co-author of a number of works on the peer review system, including a 1981 National Academy of Sciences study on its ethical aspects. (2) "That is where the principle of merit-based review was very clearly established and has been followed ever since." Cole argues that, whatever its flaws, peer review has worked. "It's been an essential part of the American science scene and one of the reasons why American science has done so well."

Peer review is like democracy was to Churchill: "the worst form of government except all
those other forms that have been tried from
time to time"

But the system that arose during decades of abundance now operates when funds and publication credits are more scarce. "A lot [of the criticism] is contingent upon the rejection rate of the journals or the grant agencies," Cole says. "If the rejection rate is 95 or 98 percent, as it is in some core journals these days, then I have no doubt whatsoever that there is an element of arbitrariness or taste. Acceptance might depend on the luck of the reviewer draw." In other words, when there are far more qualified grant proposals or articles than money or space to reward them, the distinction between winners and losers may be finer than the resolving power of human judgment. If the merits of two projects or papers are for all intents and purposes equal, a reviewer may be apt to choose between one's academic or scholarly presuppositions, biases, and personal tastes.

Dr. Eve Barak, special assistant for peer review at the NIH, says grant reviewers screen more funding requests today. And as research costs rise, the average applicant seeks more money. As a result, although research funding has increased, it hasn't kept pace with demand. "There are certainly applications that receive very favorable reviews and written commentary [that] don't make it because there isn't enough money," Barak says. Robert Moore, of the division of research grants, says that in 1985, NIH considered 20,406 projects and funded 6,752--a success rate of 33.1 percent. In 1994, NIH considered 25,510 projects, of which 6,474 received awards, for a success rate of 25.4 percent. "We're talking about the pain threshold out there," Moore says.

Against the backdrop of worsening odds for both senior and junior investigators, the General Accounting Office, an investigating body of the U.S. Congress, issued a 1994 study of peer review. GAO auditors examined 246 winning and losing grants and interviewed 1,400 reviewers at the NIH, the NSF, and the National Endowment for the Humanities. The 133-page report supported many of the complaints of critics.(3)

CHANGING THE PROCESS

THE GAO FOUND that reviewers often knew applicants and tended to give them higher scores than they would give strangers. Prestigious applicants got the benefit of the doubt over lesser-known colleagues. Applicants from top institutions enjoyed a halo effect that boosted scores. Only the NSF provided data about the race of applicants. Though non-whites did worse than whites in NSF competitions, the GAO said it lacked the evidence to draw conclusions. As to charges of gender bias, the GAO found discordant gender gaps. At NSF and NEH, men were more likely than women to get funded. At the larger NIH, on the other hand, women were funded more frequently than men.

Yet after noting these shortcomings the GAO concluded: "Overall, we found that peer review processes appear to be working reasonably well. . . ; Virtually no one suggested replacing it." The report suggested adding more women, minority members, and junior researchers to review processes, but otherwise left the system alone. Officials at NIH, NSF, and NEH all promised to take the GAO's suggestions to heart. As NSF officials wrote in reply, "We view such efforts as important for providing diverse viewpoints on the merits of the scientific or educational content of proposals, not as mere representation for its own sake."

A peer reviewer must preserve scholarly integrity by rising above the three deadly sins of intellectual life: envy, favoritism, and the temptation to plagiarize

Similarly, in 1993, when JAMA held its Second International Congress on Peer Review in Biomedical Publication, 275 participants from around the world were nearly unanimous in supporting peer review. However, they urged reforms such as blind submissions to keep the applicant's name and affiliation from influencing reviewers.

Journals in some fields have used blind review for years. Phyllis Franklin, executive director of the Modern Language Association, says her organization's literary-critical journal PMLA started removing the names of authors from refereed papers in 1980. "This blind process did seem to benefit the younger scholars and the graduate students, and to some extent the women scholars."

In a JAMA article written after the 1993 Congress, science historian Judson said electronic publication could revolutionize peer review by allowing scientists to post their work on the Internet. Many physicists already circulate their papers via electronic bulletin boards as soon as they have submitted the work to journals. "How that will affect peer review is a very interesting question," says Columbia's Cole. He says he can envision a system in which "a tremendous amount would go out on electronic format" and the community would decide, depending on citation, what would become permanent electronically or be published. "This is a reversal of what we have now, in which a small group of people, rather than the larger community, determines what becomes permanent."

While technology may one day break the bottleneck of peer review for publication, there is no easy fix for the funding scarcity that puts the grant review process under such pressure. And it is unlikely that any grant-reviewing system could do away with the need for individuals to exercise integrity. Though specific reforms are frequently suggested, even the system's harshest critics find that if existing peer review mechanisms were jettisoned entirely, the intellectual community would be hard-pressed to come up with a workable wholesale alternative.

Drummond Rennie, deputy editor of JAMA, put it this way in an article summarizing the pluses and minuses of the system. "Peer review is like democracy," he wrote, "which is, to use Churchill's phrase, the worst form of government except all those other forms that have been tried from time to time.'"(4)


FOOTNOTES
  1. Judson, Horace Freeland, "Structural Transformations of the Sciences and the End of Peer Review," Journal of the American Medical Association, 272 (July 13, 1994), 92-94.

  2. Cole, Jonathan R., and Stephen Cole, Peer Review in the NSF: Phase 2. Washington D.C.; National Academy of Sciences, 1981. Cole, Stephen, and Jonathan R. Cole, "Chance and Consensus in Peer Review," Science, 214 (1981), p. 881-86. Zuckerman, Harriett, Janathan R. Cole, and John T. Brues, The Outer Circle: Women in the Scientific Ciommunity. New Haven: Yale University Press, 1992.

  3. Peer Review: Reforms Needed to Ensure Fairness in Federal Grant Agency Selection; General Accounting Office, June 1994; document identifier gao/pemd-94-1 Peer Review; available by phone (202) 512-6000 or fax (301) 258-4066.

  4. Rennie, Drummond, "More Peering into Editorial Peer Review," Journal of the American Medical Association, 270 (Dec. 15, 1993), 2856-58.

Illustration by Ycel Erdogan

Confessions of Columbia
peer reviewers

How do scientists deal with potential conflicts of interest that arise when they review grant applications or articles? 21stC asked several Columbia scientists about their experience.

Are there tendencies not to publish papers or award grants
to colleagues you don't particularly like?

"I have a simple rule: All refereeing for journal articles is allowed to be anonymous, and about 90 percent of the time, it is. But I always sign my reviews. When you know you are going to sign your review at the end, it modifies your tone." David Helfand, Professor of Astronomy

"I take a hard line. If I am asked to review a close colleague, I decline. If I am asked to review someone who has not treated me well, I decline. Between those two boundaries, I will review my peers. I follow several other important rules: If the area of research touches on my expertise but is more broadly based, I will review only those parts that I feel competent to assess. And it's important to separate yourself from your own position. It is quite possible that someone can come up with results that contradict your own views (published or otherwise) and be right. One should be willing to accept that." Don J. Melnick, Professor of Anthropology and Biological Sciences

"I have squashed papers from people I don't like, but also from people I do like. There are objective ways of deciding. I have always removed myself from cases of grant review or panel service when the grant of someone I do not like is up for consideration. Distastes in my field are so public that you rarely need to do this, but it does happen." David Walker, chairman, Department of Geological Sciences

"There is enough peer review so that anyone who has an ax to grind takes a chance of gaining a reputation of being selfish or not fair-minded. Your credibility as a reviewer is diminished, and then your credibility as a scientist becomes diminished." Nicholas J. Turro, William P. Schweitzer Professor of Chemistry

Are there prejudices against younger investigators or older, established ones?

"Mostly, I have seen the benefit of the doubt given to young investigators. I have found that there is more carping at people who are successful. People will say, 'This guy already has $600,000 and 11 grants; why does he need another?' That's much more common." David Helfand

"These things are never perfect; there are always biases. There are networks and relationships, and somebody outside the system has less of a chance of getting a hearing than someone who knows everybody. But in my community, I do not see these problems. Certainly we try hard to be fair." Zvi Galil, Julian Clarence Levi Professor of Computer Science

How do you hold at bay any desire not to award grants to competing institutions?

"One tries to be fair and say what is right about the competition as well as what is wrong. But this is a tough situation; I know of many cases where it is a real problem." David Walker

"The desire becomes especially keen if one is competing for a limited amount of funds. Another strong institution can sweep it all up." Paul G. Richards, Mellon Professor of Natural Sciences

Are there inclinations to overcompensate and look excessively favorably on one's rivals?

"I have heard people say that it is often the scientist with the most antithetical view [to a proposal or grant] that stands there and tries to defend it. They make an extra effort to be impartial and to defend both sides. They will say, 'I personally disagree with the premise, but it is a sound premise.'" Richard J. Sohn, Director of Contracts and Grants, Health Sciences Division

"It happens all the time, surprisingly. I think it's called the hostage principle: They opposed you, they didn't wipe you out, so you end up saying quite nice things about them." Paul G. Richards

How do you resist temptation to take results from colleagues
or competitors?

"Don't do this--at least without their permission. I find it easy to hold it against someone if they propose to do something or to publish something that I've done myself and they don't acknowledge me." Paul G. Richards

"Nobody in his right mind would take results from another scientist, but there's a grayer area. If you are talking with someone over a beer or reading over a proposal and see an idea, one doesn't alway have the ability to categorize and remember these things a year later. When I get an idea, I write it down and copy it into a notebook and date it. Of course, that can be faked, but it provides a systematic and mechanical way to remember something two or three years down the pike." Nicholas J. Turro

Are there other ethical issues you've encountered?

"All the time, I find out about new sources of information, data, or funding. What, then, is my obligation to communicate this to competitors? There are many opportunities for sins of omission rather than of commission." Paul G. Richards


21stC home page 21stC is. .
. special metanews special