Evidence to Abandon Ship?

Ideally, good medical practices are replaced by better ones, based on robust comparative trials in which new treatments outperform older ones and establish new standards of care. Often, however, established standards must be abandoned not because a better replacement has been identified but simply because what was thought to be beneficial was not. In these cases, it becomes apparent that clinicians, encouraged by professional societies and guidelines, have been using medications, procedures, or preventive measures in vain.

For example, percutaneous coronary intervention performed for stable coronary artery disease and hormone therapy prescribed for postmenopausal women cost billions of dollars and supported the existence of entire specialties for many years. Stable coronary artery disease accounted for 85% of all stenting in the United States at the time of the Clinical Outcomes Utilizing Revascularization and Aggressive Drug Evaluation (COURAGE) trial. 1 Large, well-designed randomized trials that tested whether these practices improved major patient outcomes revealed that patients were not being helped. Defenders of these therapies and interventions wrote rebuttals and editorials and fought for their specialties, but the reality was that the best that could be done was to abandon ship. Often, what is a standard of practice involves the livelihood of countless specialists. How many are willing to abandon an ineffective status quo if their lifestyle is dependent upon maintaining it?

How many established standards of medical care are wrong? It is not known. Medical practice has evolved out of centuries of theorizing, personal experiences, bits of evidence, expert consensus, and diverse conflicts and biases. Rigorous questioning of long-established practices is difficult for most to undertake due to the risk of being ostracized and ridiculed by your peers, as well as being blackballed by pharmaceutical and device manufacturer research funding. There are thousands of clinical trials, but most deal with trivialities or efforts to increase the sales of specific products. The overwhelming majority of these trials are designed, controlled, and conducted by companies who stand to benefit from “positive” results.

Given this conundrum, it is likely that some entire medical subspecialties are based on very little evidence. Their disappearance probably would not harm patients and might help salvage derailed health budgets. However, it is unlikely that specialists would support trials testing practices that constitute their main source of income. Instead, the research community performs studies of minimal value without even knowing whether the basic standards of care are appropriate.

Rarely, some investigators find the courage to test established “truths” with large, rigorous randomized trials. When this happens, empirical evidence suggests that “medical reversals” can be quite common. In an evaluation of 35 trials that were published in a major clinical journal in 2009 and that tested an established clinical practice, 16 (46%) reported results consistent with current beneficial practice, 16 (46%) reported evidence that contradicted current practice and constituted a reversal, and another 3 (9%) were inconclusive. 2 Perhaps high-profile general medical journals are more prone to publish unusual results and less inclined to defend a clinical practice or specialized turf than specialty journals. However, it is unlikely that the selection filter in favor of reversal publications is stronger than the selection filter favoring the validation of standard of care.

The mere testing of a standard of care generates interest because many standards of care are never tested. In another evaluation of trials published in 3 major general medical journals or high–impact factor specialty journals, 3 of the 39 most cited randomized trials published in 1990-2003 that found a significant benefit of a clinical intervention, 9 (23%) found effects stronger than those found in subsequent studies and 19 (49%) found results replicated in subsequent studies, but 11 (28%) remained largely unchallenged, with no large trial conducted on the same question. 3

Many medical reversals involve conditions for which the standard of care has been promoted over the years based primarily on pathophysiological considerations. Often one or more trials exist, but they have not tested clinically relevant outcomes or have been biased. For example, vertebroplasty—the injection of polymethylmethacrylate cement into fractured bone—gained popularity in the early 2000s for the treatment of osteoporotic fractures. Initial studies addressed the pathophysiology of this therapy, delineated the technical skills required to optimally perform the procedure, and furthered the discussion about the benefits of vertebroplasty. Claims of benefit were strongly contradicted in 2 randomized trials 4,5 that included a sham (placebo) procedure, which alone might have been responsible for pain relief. Trials without sham (placebo) control might continue to show benefit, but it is difficult to justify performing invasive, expensive operations simply to obtain placebo effects.

Despite the evidence, many specialists will not abandon the procedure. A study of vertebroplasty utilization at one in- stitution showed little reduction in referrals after publication of studies contradicting current practice and in fact even showed that increasing percentages of referred patients were offered the procedure.6 Similarly, the results of COURAGE have done little to improve optimal medical management of stable coronary artery disease prior to invasive intervention. Stenting may not improve mortality, but the procedure apparently diminishes angina. 7 However, more than 50 years ago, Cobb et al 8 demonstrated that large improvements in pain with internal mammary artery ligation were comparable to results obtained with a sham (placebo) procedure. As is the case with vertebroplasty, stenting performed in patients with stable disease is probably widely used as an expensive placebo for pain control.

The increasing use of surrogate end points and short-term outcomes has also affected the credibility of clinical trial results. Reversal of important research findings is more common when pragmatic clinical outcomes are not used for initial approval and licensing of interventions. For instance, bevacizumab exploited an accelerated approval process for treatment of metastatic breast cancer. Approval was granted based on preliminary data on disease progression, an end point that may not necessarily translate into improved life expectancy or quality of life for patients. After accrual of further data, on November 18, 2011, the US Food and Drug Administration revoked its prior approval for this indication. Yet it only took weak data for approval! In other words, patients paid to be guinea pigs

Because medicine is in part a statistically driven science, a certain amount of reversal of standards of care is inevitable. However, what is currently tolerated is far greater than the uncertainty of statistics. Reversal of established practices implies at least 3 grave consequences besides unjustified cost. First, patients who undergo the therapy during the years it is in favor receive all the risk of treatment and, ultimately, no real benefit. Second, contradicting studies do not immediately force a change in practice; the contradicted practice continues for years. 9 Third, contradiction of mainstream practices undermines trust in the medical system, and shows the cracks in the foundation of “modern medicine”. The implications of reversal are significant. Reversal implies error or harm to patients who underwent the practice in question, during the years it was considered effective, opening the doors for catastrophic litigation.

Given the slow rate of abandonment of ineffective medical practices, the standards governing drug and device approval must be strengthened. This means that newly proposed innovations should be evaluated in sufficiently large unbiased randomized trials that demonstrate improvement in important clinical end points before being widely disseminated.

Such an insistence on well-designed, large studies may be seen as overly costly during times of financial hardship. However, the costs of permitting increasing widespread use of ineffective interventions like chemotherapy are much greater.

In the case of vertebroplasty, a few million dollars used to conduct a proper clinical trial before regulatory approval might have saved nearly a billion dollars a year over the course of a decade. 10 For unnecessary hormone therapy and coronary stenting, the cost has been even greater. Large trials of new innovations should be designed and conducted by investigators without conflicts of interest, under the auspices of nonconflicted scientific bodies. Instead of designing, controlling, and conducting the trials, manufacturers may offer the respective budget to a centralized public pool of funding, keeping the trial design and conduct independent.

Realistically though, asking corporate sponsors to perform critical trials on their own products is like asking a painter to judge their own painting so as to receive an award. If a manufacturer can be allowed to manipulate the system to create a blockbuster product from an ineffective drug, the temptation is hard to resist.

Besides the need for better evidence for new interventions, medical reversals also suggest that reality checks should be encouraged for established practices that constitute the core of medical care. Priority should be given to testing practices having limited or no prior randomized evidence for their use, reassessing old evidence that may no longer be relevant for current clinical settings, and evaluating therapies and interventions that are most expensive.

Sometimes the beliefs of Doctors can have similar psychological characteristics as the nonscientific beliefs observed in other areas of society. The wish bias of individuals, irrespective of topic, can be large and may also influence the interpretation of scientific results and lead to the precribing of ineffective protocols due to ego and “wishful thinking”. Such bias has been discussed and demonstrated in the past for several other societal and scientific efforts. 11-16

Wish bias does not necessarily mean that the defended beliefs are wrong. Moreover, it can be difficult to discern whether perpetuated beliefs are based on careful consideration of all evidence and differential interpretation, inappropriate entrenchment of old information, lack of dissemination of newer data, or purposeful silencing of their existence. Regardless of the reasons, better communication between research specialists and evidence-based clinical science 17 may improve this situation and may lead to more rational and concerted translational efforts in basic, preclinical, and clinical research.


1. Boden WE, O’Rourke RA, Teo KK, et al; COURAGE Trial Research Group. Optimal medical therapy with or without PCI for stable coronary disease. N Engl J Med. 2007;356(15):1503-1516.
2. Prasad V, Gall V, Cifu A. The frequency of medical reversal. Arch Intern Med. 2011;171(18):1675-1676.
3. Ioannidis JP. Contradicted and initially stronger effects in highly cited clinical research. JAMA . 2005;294(2):218-228.
4. Buchbinder R, Osborne RH, Ebeling PR, et al. A randomized trial of vertebroplasty for painful osteoporotic vertebral fractures. N Engl J Med. 2009;361 (6):557-568.
5. Kallmes DF, Comstock BA, Heagerty PJ, et al. A randomized trial of vertebroplasty for osteoporotic spinal fractures. N Engl J Med. 2009;361(6):569-579.
6. Luetmer MT, Kallmes DF. Have referral patterns for vertebroplasty changed since publication of the placebo-controlled trials? AJNR Am J Neuroradiol. 2011; 32(4):647-648.
7. Weintraub WS, Spertus JA, Kolm P, et al; COURAGE Trial Research Group. Effect of PCI on quality of life in patients with stable coronary disease. N Engl J Med. 2008;359(7):677-687.
8. Cobb LA, Thomas GI, Dillard DH, Merendino KA, Bruce RA. An evaluation of internal-mammary-artery ligation by a double-blind technic. N Engl J Med. 1959; 260(22):1115-1118.
9. Tatsioni A, Bonitsis NG, Ioannidis JPA. Persistence of contradicted claims in the literature. JAMA. 2007;298(21):2517-2526.
10. Elshaug AG, Garber AM. How CER could pay for itself—insights from vertebral fracture treatments. N Engl J Med. 2011;364(15):1390-1393.
11. Markovits H, Nantel G. The belief-bias effect in the production and evaluation of logical conclusions. Mem Cognit. 1989;17(1):11-17.
12. Kaptchuk TJ. Effect of interpretive bias on research evidence. BMJ. 2003;326(7404):1453-1455.
13. Koehler JJ. The influence of prior beliefs on scientific judgements of evidence quality. Organ Behav Hum Decis Process. 1993;56:28-55.
14. Resch KI, Ernst E, Garrow J. A randomized controlled study of reviewer bias against an unconventional therapy. J R Soc Med. 2000;93(4):164-167.
15. MacCoun RJ. Biases in the interpretation and use of research results. Annu Rev Psychol. 1998;49:259-287.
16. Bazerman MH, Loewenstein G, Moore DA. Why good accountants do bad audits. Harv Bus Rev. 2002; 80(11):97-102.
17. Ioannidis JP. Evolution and translation of research findings: from bench to where? PLoS Clin Trials. 2006;1(7):e36.

mammograms-and-nejm-big-studyMammography exposed in the NEJM

mammograms-cause-breast-cancer-dr-ben-johnsonMammograms cause Breast Cancer

mammograms-do-not-save-lives-cnnMammograms do not save lives CNN

mammograms-get-some-women-unnecessary-treatmentMammograms get many women unnecessary treatment

mammograms-save-lives-debunked-nbc-newsMammograms Save Lives? Debunked on NBC News

mammogram-study_test-doesnt-save-lives-of-women-40-59Mammogram study: Test doesn’t save lives of women 40-59

mammogram-the-history-and-the-shamMammogram: the history and the sham

mammography-is-ineffective-ctv-national-newsMammography is ineffective, CTV National News

mammography-the-truth-about-itMammography: the truth about it