E T H I C S I N R E S E A R C H
What's the verdict
on peer review?
The judgment
of colleagues plays a critical part
in how grants are distributed, journal articles are
selected, and careers are formed. Yet this system may
raise ethical dilemmas.
21stC explores how peer review
really works, and how it might work better
By TOM ABATE
A YOUNG CARDIOLOGIC researcher published his findings in the world's most prestigious medical journal-which later retracted the paper when auditors found that he had cooked his data. An investigator studying drug treatment for hyperactive children did likewise. A top American immunologist was chastened for failing to adequately credit overseas colleagues in the identification of a new virus. And one of America's foremost scientists, at first supported by deferential peers against a largely defenseless junior colleague in a dispute over the publication of questionable data, was eventually forced to resign his prestigious post when the data were determined to be bogus. Why didn't the peer review system prevent these instances of unethical behavior? Was the system, as many have argued, part of the problem?The ethical foundations of the peer review process have come under heated questioning in recent years. Women and minorities have charged that an old boys' network denies them fair due. Smaller or relatively obscure institutions feel slighted by larger rivals. And young scientists have complained that the system stacks the deck against their work, even though their whole careers are at stake. (Particularly troublesome for the younger investigator is the Oz-behind-the-curtain effect: Reviewers often work anonymously, giving them greater opportunity to act arbitrarily. The reviewee usually has no comparable curtain to stand behind.) Even Congress has investigated peer review, spurred by the well-publicized frauds and coverups of the 1980s, which the ostensibly self-policing mechanisms known as peer review had failed to disclose or prevent.
From 1985 to 1994, NIH's success rate dropped
from a third to a quarter. "We're talking about the pain threshold"
Strictly speaking, peer review describes various processes by which panels of experts read and
grade grant applications, issuing opinions that influence funding decisions. Reviewing articles
for journal publication is more accurately called refereeing, but the term "peer review" describes
this editorial process as well. All these activities pose ethical challenges. Whether grading
grants or screening articles, a peer reviewer must preserve scholarly integrity by rising above the
three deadly sins of intellectual life: envy, favoritism, and the temptation to plagiarize. For, as
science historian Horace Freeland Judson observed last year in the Journal of the
American Medical Association (JAMA): . . .the persons most qualified to judge the
worth of a scientist's grant proposal or the merit of a submitted research paper are precisely those
who are the scientist's closest competitors." (1)
PAST AND PRESENT PRACTICES
HOW DID PEER REVIEW, with all its built-in conflicts, become the judicial system of the
intellectual world? Michael Crowe, a science historian at the University of Notre
Dame, says that by the 1840s, the Royal Society in London was occasionally asking
scientists to read papers submitted for publication in its Philosophical Transactions.
Sally Gregory Kohlstedt, historian of science at the University of Minnesota, says editors of early American
journals had the same informal attitude toward peer review, using it only when they lacked the
expertise to judge submissions. Peer review took a tentative step into the realm of funding after
1863, when the newly formed National Academy of
Sciences began receiving private gifts to support research. It formed ad hoc committees of
experts to distribute the awards. But Kohlstedt said not until World War I, when the federal
government began supporting scientists through the National Research Council and questions arose
about fair distribution of a public resource, did these private committees start to bear the burden
that peer review carries today. "I don't think ethics played much of a role until the 20th century,
when more money became available from private and public pockets," Kohlstedt says.
Thus, by the post-World War II science boom, peer review had become accepted practice. "It
came into full force after the war with the establishments of the National Science Foundation and
the National Institutes of Health," says
Jonathan R. Cole, provost of Columbia and co-author of a number of works on the peer review
system, including a 1981 National Academy of Sciences study on its ethical aspects. (2) "That is where the principle of merit-based review was very
clearly established and has been followed ever since." Cole argues that, whatever its flaws, peer
review has worked. "It's been an essential part of the American science scene and one of the
reasons why American science has done so well."
Peer review is like democracy was to Churchill: "the worst form of government except all
But the system that arose during decades of abundance now operates when funds and publication
credits are more scarce. "A lot [of the criticism] is contingent upon the rejection rate of the
journals or the grant agencies," Cole says. "If the rejection rate is 95 or 98 percent, as it is in
some core journals these days, then I have no doubt whatsoever that there is an element of
arbitrariness or taste. Acceptance might depend on the luck of the reviewer draw." In other
words, when there are far more qualified grant proposals or articles than money or space to
reward them, the distinction between winners and losers may be finer than the resolving power of
human judgment. If the merits of two projects or papers are for all intents and purposes equal, a
reviewer may be apt to choose between one's academic or scholarly presuppositions, biases, and
personal tastes.
Dr. Eve Barak, special assistant for peer review at the NIH, says grant reviewers screen more
funding requests today. And as research costs rise, the average applicant seeks more money. As a
result, although research funding has increased, it hasn't kept pace with demand. "There are
certainly applications that receive very favorable reviews and written commentary [that] don't
make it because there isn't enough money," Barak says. Robert Moore, of the division of research
grants, says that in 1985, NIH considered 20,406 projects and funded 6,752--a success rate of
33.1 percent. In 1994, NIH considered 25,510 projects, of which 6,474 received awards, for a
success rate of 25.4 percent. "We're talking about the pain threshold out there," Moore says.
Against the backdrop of worsening odds for both senior and junior investigators, the General Accounting Office, an investigating body of the
U.S. Congress, issued a 1994 study of peer review. GAO auditors examined 246 winning and
losing grants and interviewed 1,400 reviewers at the NIH, the NSF, and the National Endowment for the Humanities. The 133-page
report supported many of the complaints of critics.(3)
CHANGING THE PROCESS
THE GAO FOUND that reviewers often knew applicants and tended to give them higher scores
than they would give strangers. Prestigious applicants got the benefit of the doubt over
lesser-known colleagues. Applicants from top institutions enjoyed a halo effect that boosted
scores. Only the NSF provided data about the race of applicants. Though non-whites did worse
than whites in NSF competitions, the GAO said it lacked the evidence to draw conclusions. As to
charges of gender bias, the GAO found discordant gender gaps. At NSF and NEH, men were
more likely than women to get funded. At the larger NIH, on the other hand, women were funded
more frequently than men.
Yet after noting these shortcomings the GAO concluded: "Overall, we found that peer review
processes appear to be working reasonably well. . . ; Virtually no one suggested replacing it."
The report suggested adding more women, minority members, and junior researchers to review
processes, but otherwise left the system alone. Officials at NIH, NSF, and NEH all promised to
take the GAO's suggestions to heart. As NSF officials wrote in reply, "We view such efforts as
important for providing diverse viewpoints on the merits of the scientific or educational content
of proposals, not as mere representation for its own sake."
A peer reviewer must preserve scholarly integrity by rising above the three deadly sins of
intellectual life: envy, favoritism, and the temptation to plagiarize
Similarly, in 1993, when JAMA held its Second International Congress on Peer
Review in Biomedical Publication, 275 participants from around the world were nearly
unanimous in supporting peer review. However, they urged reforms such as blind submissions
to keep the applicant's name and affiliation from influencing reviewers.
Journals in some fields have used blind review for years. Phyllis Franklin, executive director of
the Modern Language Association, says her organization's literary-critical journal
PMLA started removing the names of authors from refereed papers in 1980. "This
blind process did seem to benefit the younger scholars and the graduate students, and to some
extent the women scholars."
In a JAMA article written after the 1993 Congress, science historian Judson said
electronic publication could revolutionize peer review by allowing scientists to post their work
on the Internet. Many physicists already circulate their papers via electronic bulletin boards as
soon as they have submitted the work to journals. "How that will affect peer review is a very
interesting question," says Columbia's Cole. He says he can envision a system in which "a
tremendous amount would go out on electronic format" and the community would decide,
depending on citation, what would become permanent electronically or be published. "This is a
reversal of what we have now, in which a small group of people, rather than the larger
community, determines what becomes permanent."
While technology may one day break the bottleneck of peer review for publication, there is no
easy fix for the funding scarcity that puts the grant review process under such pressure. And it is
unlikely that any grant-reviewing system could do away with the need for individuals to exercise
integrity. Though specific reforms are frequently suggested, even the system's harshest critics
find that if existing peer review mechanisms were jettisoned entirely, the intellectual community
would be hard-pressed to come up with a workable wholesale alternative.
Drummond Rennie, deputy editor of JAMA, put it this way in an article
summarizing the pluses and minuses of the system. "Peer review is like democracy," he wrote,
"which is, to use Churchill's phrase, the worst form of government except all those other forms
that have been tried from time to time.'"(4)
Are there tendencies not to publish papers or award grants
"I have a simple rule: All refereeing for journal articles is allowed to be anonymous, and about
90 percent of the time, it is. But I always sign my reviews. When you know you are going to sign
your review at the end, it modifies your tone." David Helfand, Professor of
Astronomy
"I take a hard line. If I am asked to review a close colleague, I decline. If I am asked to review
someone who has not treated me well, I decline. Between those two boundaries, I will review my
peers. I follow several other important rules: If the area of research touches on my expertise but
is more broadly based, I will review only those parts that I feel competent to assess. And it's
important to separate yourself from your own position. It is quite possible that someone can
come up with results that contradict your own views (published or otherwise) and be right. One
should be willing to accept that." Don J. Melnick, Professor of Anthropology and
Biological Sciences
"I have squashed papers from people I don't like, but also from people I do like. There are
objective ways of deciding. I have always removed myself from cases of grant review or panel
service when the grant of someone I do not like is up for consideration. Distastes in my field are
so public that you rarely need to do this, but it does happen." David Walker, chairman,
Department of Geological Sciences
"There is enough peer review so that anyone who has an ax to grind takes a chance of gaining a
reputation of being selfish or not fair-minded. Your credibility as a reviewer is diminished, and
then your credibility as a scientist becomes diminished." Nicholas J. Turro, William P.
Schweitzer Professor of Chemistry
Are there prejudices against younger investigators or older, established
ones?
"Mostly, I have seen the benefit of the doubt given to young investigators. I have found that
there is more carping at people who are successful. People will say, 'This guy already has
$600,000 and 11 grants; why does he need another?' That's much more common." David
Helfand
"These things are never perfect; there are always biases. There are networks and relationships,
and somebody outside the system has less of a chance of getting a hearing than someone who
knows everybody. But in my community, I do not see these problems. Certainly we try hard to
be fair." Zvi Galil, Julian Clarence Levi Professor of Computer Science
How do you hold at bay any desire not to award grants to competing institutions?
"One tries to be fair and say what is right about the competition as well as what is wrong. But
this is a tough situation; I know of many cases where it is a real problem." David
Walker
"The desire becomes especially keen if one is competing for a limited amount of funds. Another
strong institution can sweep it all up." Paul G. Richards, Mellon Professor of Natural
Sciences
Are there inclinations to overcompensate and look excessively favorably on one's
rivals?
"I have heard people say that it is often the scientist with the most antithetical view [to a proposal
or grant] that stands there and tries to defend it. They make an extra effort to be impartial and to
defend both sides. They will say, 'I personally disagree with the premise, but it is a sound
premise.'" Richard J. Sohn, Director of Contracts and Grants, Health Sciences
Division
"It happens all the time, surprisingly. I think it's called the hostage principle: They opposed you,
they didn't wipe you out, so you end up saying quite nice things about them." Paul G.
Richards
How do you resist temptation to take results from colleagues
"Don't do this--at least without their permission. I find it easy to hold it against someone if they
propose to do something or to publish something that I've done myself and they don't
acknowledge me." Paul G. Richards
"Nobody in his right mind would take results from another scientist, but there's a grayer area. If
you are talking with someone over a beer or reading over a proposal and see an idea, one doesn't
alway have the ability to categorize and remember these things a year later. When I get an idea, I
write it down and copy it into a notebook and date it. Of course, that can be faked, but it provides
a systematic and mechanical way to remember something two or three years down the pike."
Nicholas J. Turro
Are there other ethical issues you've
encountered?
"All the time, I find out about new sources of information, data, or funding. What, then, is my
obligation to communicate this to competitors? There are many opportunities for sins of
omission rather than of commission." Paul G. Richards
those other forms that have been tried from
time to time"
FOOTNOTES
Illustration by Ycel Erdogan
Confessions of Columbia
How do scientists deal with potential conflicts of interest that arise when they review grant
applications or articles? 21stC asked several Columbia scientists about their
experience.
peer reviewers
to colleagues you don't
particularly like?
or competitors?
