Altman, Lawrence K. "For Science's Gatekeepers, A Credibility Gap," New York Times, 2

 

Altman, Lawrence K. "For Science's Gatekeepers, A Credibility Gap," New York Times, 2
May 2006, pp.F1-F7.

Recent disclosures of fraudulent or flawed studies in medical and scientific journals have called
into question as never before the merits of their peer-review system.

The system is based on journals inviting independent experts to critique submitted manuscripts.
The stated aim is to weed out sloppy and bad research, ensuring the integrity of the what it has
published.

Because findings published in peer-reviewed journals affect patient care, public policy and the
authors' academic promotions, journal editors contend that new scientific information should be
published in a peer-reviewed journal before it is presented to doctors and the public.

That message, however, has created a widespread misimpression that passing peer review is the
scientific equivalent of the Good Housekeeping seal of approval.

Virtually every major scientific and medical journal has been humbled recently by publishing
findings that are later discredited. The flurry of episodes has led many people to ask why authors,
editors and independent expert reviewers all failed to detect the problems before publication.

The publication process is complex. Many factors can allow error, even fraud, to slip through.
They include economic pressures for journals to avoid investigating suspected errors; the desire
to avoid displeasing the authors and the experts who review manuscripts; and the fear that angry
scientists will withhold the manuscripts that are the lifeline of the journals, putting them out of
business. By promoting the sanctity of peer review and using it to justify a number of their
actions in recent years, journals have added to their enormous power.

The release of news about scientific and medical findings is among the most tightly managed in
country. Journals control when the public learns about findings from taxpayer-supported research
by setting dates when the research can be published. They also impose severe restrictions on
what authors can say publicly, even before they submit a manuscript, and they have penalized
authors for infractions by refusing to publish their papers. Exceptions are made for scientific
meetings and health emergencies.

But many authors have still withheld information for fear that journals would pull their papers for
an infraction. Increasingly, journals and authors' institutions also send out news releases ahead of
time about a peer-reviewed discovery so that reports from news organizations coincide with a
journal's date of issue.

A barrage of news reports can follow. But often the news release is sent without the full paper, so
reports may be based only on the spin created by a journal or an institution.

Journal editors say publicity about corrections and retractions distorts and erodes confidence in
science, which is an honorable business. Editors also say they are gatekeepers, not detectives, and
that even though peer review is not intended to detect fraud, it catches flawed research and
improves the quality of the thousands of published papers.

However, even the system's most ardent supporters acknowledge that peer review does not
eliminate mediocre and inferior papers and has never passed the very test for which it is used.
Studies have found that journals publish findings based on sloppy statistics. If peer review were a
drug, it would never be marketed, say critics, including journal editors.

None of the recent flawed studies have been as humiliating as an article in 1972 in the journal
Pediatrics that labeled sudden infant death syndrome a hereditary disorder, when, in the case
examined, the real cause was murder.

Twenty-three years later, the mother was convicted of smothering her five children. Scientific
naïveté surely contributed to the false conclusion, but a forensic pathologist was not one of the
reviewers. The faulty research in part prompted the National Institutes of Health to spend
millions of dollars on a wrong line of research.

Fraud, flawed articles and corrections have haunted general interest news organizations. But such
problems are far more embarrassing for scientific journals because of their claims for the
superiority of their system of editing.

A widespread belief among nonscientists is that journal editors and their reviewers check
authors' research firsthand and even repeat the research. In fact, journal editors do not routinely
examine authors' scientific notebooks. Instead, they rely on peer reviewers' criticisms, which are
based on the information submitted by the authors.

While editors and reviewers may ask authors for more information, journals and their invited
experts examine raw data only under the most unusual circumstances.

In that respect, journal editors are like newspaper editors, who check the content of reporters'
copy for facts and internal inconsistencies but generally not their notes. Still, journal editors have
refused to call peer review what many others say it is — a form of vetting or technical editing.

In spot checks, many scientists and nonscientists said they believed that editors decided what to
publish by counting reviewers' votes. But journal editors say that they are not tally clerks and
that decisions to publish are theirs, not the reviewers'.

Editors say they have accepted a number of papers that reviewers have harshly criticized as
unworthy of publication and have rejected many that received high plaudits.

Many nonscientists perceive reviewers to be impartial. But the reviewers, called independent
experts, in fact are often competitors of the authors of the papers they scrutinize, raising potential
conflicts of interest.

Except when gaffes are publicized, there is little scrutiny of the quality of what journals publish.

Journals have rejected calls to make the process scientific by conducting random audits like those
used to monitor quality control in medicine. The costs and the potential for creating distrust are
the most commonly cited reasons for not auditing.

In defending themselves, journal editors often shift blame to the authors and excuse themselves
and their peer reviewers.

Journals seldom investigate frauds that they have published, contending that they are not
investigative bodies and that they could not afford the costs. Instead, the journals say that the
investigations are up to the accused authors' employers and agencies that financed the research.

Editors also insist that science corrects its errors. But corrections often require whistle-blowers or
prodding by lawyers. Editors at The New England Journal of Medicine said they would not have
learned about a problem that led them to publish two letters of concern about omission of data
concerning the arthritis drug Vioxx unless lawyers for the drug's manufacturer, Merck, had asked
them questions in depositions. Fraud has also slipped through in part because editors have long
been loath to question the authors.

"A request from an editor for primary data to support the honesty of an author's findings in a
manuscript under review would probably poison the air and make civil discourse between
authors and editors even more difficult than it is now," Dr. Arnold S. Relman wrote in 1983. At
the time, he was editor of The New England Journal of Medicine, and it had published a
fraudulent paper.

Fraud is a substantial problem, and the attitude toward it has changed little over the years, other
editors say. Some journals fail to retract known cases of fraud for fear of lawsuits.

Journals have no widely accepted way to retract papers, said Donald Kennedy, editor in chief of
Science, after the it retracted two papers by the South Korean researcher Dr. Hwang Woo Suk,
who fabricated evidence that he had cloned human cells.

In the April 18 issue of Annals of Internal Medicine, its editor, Dr. Harold C. Sox, wrote about
lessons learned after the journal retracted an article on menopause by Dr. Eric Poehlman of the
University of Vermont.

When an author is found to have fabricated data in one paper, scientists rarely examine all of that
author's publications, so the scientific literature may be more polluted than believed, Dr. Sox
said.

Dr. Sox and other scientists have documented that invalid work is not effectively purged from the
scientific literature because the authors of new papers continue to cite retracted ones.

When journals try to retract discredited papers, Dr. Sox said, the process is slow, and the system
used to inform readers faulty. Authors often use euphemisms instead of the words "fabrication"
or "research misconduct," and finding published retractions can be costly because some affected
journals charge readers a fee to visit their Web sites to learn about them, Dr. Sox said.

Despite its flaws, scientists favor the system in part because they need to publish or perish. The
institutions where the scientists work and the private and government agencies that pay for their
grants seek publicity in their eagerness to show financial backers results for their efforts.

The public and many scientists tend to overlook the journals' economic benefits that stem from
linking their embargo policies to peer review. Some journals are owned by private for-profit
companies, while others are owned by professional societies that rely on income from the
journals. The costs of running journals are low because authors and reviewers are generally not
paid.

A few journals that not long ago measured profits in the tens of thousands of dollars a year now
make millions, according to at least three editors who agreed to discuss finances only if granted
anonymity, because they were not authorized to speak about finances.

Any influential system that profits from taxpayer-financed research should be held publicly
accountable for how the revenues are spent. Journals generally decline to disclose such data.

Although editors of some journals say they demand statements from their editing staff members
that they have no financial conflicts of interest, there is no way to be sure. At least one editor of a
leading American journal had to resign because of conflicts of interest with industry.

Journals have devolved into information-laundering operations for the pharmaceutical industry,
say Dr. Richard Smith, the former editor of BMJ, the British medical journal, and Dr. Richard
Horton, the editor of The Lancet, also based in Britain.

The journals rely on revenues from industry advertisements. But because journals also profit
handsomely by selling drug companies reprints of articles reporting findings from large clinical
trials involving their products, editors may "face a frighteningly stark conflict of interest" in
deciding whether to publish such a study, Dr. Smith said.