Flawed medical studies can end up in doctors’ advice. We developed a tool to stop it
- Written by Aya Mousa, Senior Research Fellow in Women's Cardiometabolic Health, Monash University
Good health care depends on evidence-based clinical practice guidelines. They translate the best available research into recommendations that shape diagnosis, treatment, and prevention strategies.
But what happens when the studies underpinning these guidelines are flawed?
Evidence suggests[1] scientific misconduct – from fabricated or manipulated data to methodological errors and ethical concerns – is a growing problem. In some disciplines, estimates[2] suggest as many as 40% of studies included in systematic reviews may have issues with their integrity.
This is not just an academic issue. When flawed studies are used to guide real-world health care, the consequences for health-care providers and ultimately patients can be serious. They include unnecessary or even harmful treatments, delay or denial of other effective treatments, wasted resources and a loss of public trust in science and health care itself.
Yet until recently, there has been no formal method to identify and manage flawed studies, before they make their way into clinical recommendations. We recently helped develop a framework[3] that addresses this crucial gap. Published in The Lancet’s eClinicalMedicine[4], this framework provides a step-by-step process for evaluating the integrity of studies used in clinical guidelines and systematic reviews.
In an era of increasing concern about research misconduct, it’s a timely and much-needed advance.
Clinical care relies on research integrity
Randomised controlled trials are considered the gold standard in medical research.
Their results often underpin clinical guidelines that shape day-to-day decisions in health care. But what if a randomised controlled trial contains fabricated data? Or is conducted without ethics approval? Or is retracted after being used in a previous guideline?
A 2020 study[5] found 44% of randomised controlled trials submitted to a major medical journal between 2017 and 2020 contained problematic or false data.
Compounding the problem is the fact that journal editors and publishers can be very slow to respond to concerns about research integrity.
For example, between November 2017 and April 2024, a group of researchers wrote to editors and publishers[6] of 891 potentially untrustworthy papers published in 206 different journals. At the time their study was published earlier this year, only 30% of the papers they flagged had received an outcome – 58% of which were retracted.
Notably, it took a median time of 38 months for editors and publishers to make a decision. In only 13% of the flagged cases was a decision made within 12 months.
The ripple effects of this can be enormous. A review[7] by the independent Cochrane Collaboration of nutrition interventions in pregnancy found that removing studies with integrity concerns changed the conclusions of 72% of reviews. One third (33%) needed to be updated because their guidance was no longer reliable.
Integrity concerns vary across fields. But some, such as complementary therapies or supplements[8], can be particularly prone to these concerns.
Despite this, most guideline development tools — including those from the World Health Organization[9] — assess methodological quality, not the trustworthiness or integrity of the studies that are included.
A practical framework for safeguarding integrity
Our framework[11] features a six-step process for safeguarding research integrity:
- Review: conduct a standard systematic review to identify eligible studies
- Exclude: remove studies that have been formally retracted or are flagged with serious concerns
- Assess: use available tools and checklists to assess the integrity of the remaining studies
- Discuss: convene an independent integrity committee to review ratings and vote on each study
- Establish contact: reach out to authors of high-risk studies to clarify issues or provide missing information
- Reassess: based on responses (or lack thereof), determine whether a study should be included, excluded, or held in limbo.
The integrity committee is central to this approach. It is a multidisciplinary group responsible for assessing studies objectively, without preconceived judgements or biases around which studies to exclude.
Applying the framework to the real world
Our framework was developed alongside the international evidence-based guideline[12] for polycystic ovary syndrome.
Polycystic ovary syndrome is a common hormonal, reproductive and metabolic condition affecting 8–13% of women of reproductive age[13], depending on the diagnostic criteria used. It can cause irregular menstrual cycles, elevated androgen levels, and an increased number of small follicles in the ovaries, visible on ultrasound. It is a leading cause of infertility[14].
The guideline was developed with input from diverse professional and consumer groups. It was endorsed by 39 organisations across six continents.
In making recommendations on infertility treatment in polycystic ovary syndrome, 101 studies were initially identified. After applying our framework, 45 studies were not included due to concerns about integrity. Only three authors responded to clarification requests. This illustrates the problem with transparency after publication.
Without our framework, these problematic studies may have directly shaped recommendations and health care for women with polycystic ovary syndrome around the world.
Our framework was incorporated into the National Health and Medical Research Council review process that approved the guidelines. It has since been applied to other guidelines in women’s health. Further scale up is planned.
Some drawbacks
While our framework offers a much-needed solution, it’s not without drawbacks.
First, the tools it relies on — such as a checklist for measuring trustworthiness in randomised controlled trials[16] and the research integrity assessment tool[17] — are still being refined. They also need to be validated across different research contexts.
Second, older studies (conducted before trial registries were common) or those from countries with different ethics standards, may be unfairly penalised.
There is also a risk that valid research could be excluded simply because authors do not respond to integrity enquiries.
Implementing the framework can also take time. In resource-limited settings, this may be a barrier.
But failing to assess integrity will likely cost more in the long run. It could lead to flawed recommendations, misplaced public confidence and patient harm.
References
- ^ Evidence suggests (theconversation.com)
- ^ estimates (pmc.ncbi.nlm.nih.gov)
- ^ framework (www.thelancet.com)
- ^ eClinicalMedicine (www.thelancet.com)
- ^ A 2020 study (associationofanaesthetists-publications.onlinelibrary.wiley.com)
- ^ a group of researchers wrote to editors and publishers (pubmed.ncbi.nlm.nih.gov)
- ^ review (onlinelibrary.wiley.com)
- ^ complementary therapies or supplements (pmc.ncbi.nlm.nih.gov)
- ^ World Health Organization (www.who.int)
- ^ Yuri A/Shutterstock (www.shutterstock.com)
- ^ framework (www.thelancet.com)
- ^ international evidence-based guideline (academic.oup.com)
- ^ affecting 8–13% of women of reproductive age (pubmed.ncbi.nlm.nih.gov)
- ^ leading cause of infertility (theconversation.com)
- ^ T.Schneider/Shutterstock (www.shutterstock.com)
- ^ a checklist for measuring trustworthiness in randomised controlled trials (researchintegrityjournal.biomedcentral.com)
- ^ research integrity assessment tool (onlinelibrary.wiley.com)
