The Times Australia
The Times World News

.
The Times Real Estate

.

How small differences in data analysis make huge differences in results

  • Written by Hannah Fraser, Postdoctoral Researcher , The University of Melbourne
how small differences in data analysis make huge differences in results

Over the past 20 years or so, there has been growing concern that many results published in scientific journals can’t be reproduced[1].

Depending on the field of research, studies have found efforts to redo published studies lead to different results in between 23%[2] and 89%[3] of cases.

To understand how different researchers might arrive at different results, we asked hundreds of ecologists and evolutionary biologists to answer two questions by analysing given sets of data. They arrived at a huge range of answers.

Our study has been accepted by BMC Biology as a stage 1 registered report[4] and is currently available as a preprint[5] ahead of peer review for stage 2.

Why is reproducibility a problem?

The causes of problems with reproducibility[6] are common across science. They include an over-reliance on simplistic measures of “statistical significance” rather than nuanced evaluations, the fact journals prefer to publish “exciting” findings, and questionable research practices[7] that make articles more exciting at the expense of transparency and increase the rate of false results in the literature.

Much of the research on reproducibility and ways it can be improved (such as “open science” initiatives[8]) has been slow to spread between different fields of science.

Read more: Our survey found 'questionable research practices' by ecologists and biologists – here's what that means[9]

Interest in these ideas has been growing among ecologists[10], but so far there has been little research evaluating replicability in ecology. One reason for this is the difficulty of disentangling environmental differences from the influence of researchers’ choices.

One way to get at the replicability of ecological research, separate from environmental effects, is to focus on what happens after the data is collected.

Birds and siblings, grass and seedlings

We were inspired by work led by Raphael Silberzahn[11] which asked social scientists to analyse a dataset to determine whether soccer players’ skin tone predicted the number of red cards they received. The study found a wide range of results.

We emulated this approach in ecology and evolutionary biology with an open call to help us answer two research questions:

  • “To what extent is the growth of nestling blue tits (Cyanistes caeruleus) influenced by competition with siblings?”

  • “How does grass cover influence Eucalyptus spp. seedling recruitment?” (“Eucalyptus spp. seedling recruitment” means how many seedlings of trees from the genus Eucalyptus there are.)

A photo of eucalyptus seedlings outdoors
Researchers disagreed over whether grass cover encourages or discourages Eucalyptus seedlings. Shutterstock[12]

Two hundred and forty-six ecologists and evolutionary biologists answered our call. Some worked alone and some in teams, producing 137 written descriptions of their overall answer to the research questions (alongside numeric results). These answers varied substantially for both datasets.

Looking at the effect of grass cover on the number of Eucalyptus seedlings, we had 63 responses. Eighteen described a negative effect (more grass means fewer seedlings), 31 described no effect, six teams described a positive effect (more grass means more seedlings), and eight described a mixed effect (some analyses found positive effects and some found negative effects).

For the effect of sibling competition on blue tit growth, we had 74 responses. Sixty-four teams described a negative effect (more competition means slower growth, though only 37 of these teams thought this negative effect was conclusive), five described no effect, and five described a mixed effect.

What the results mean

Perhaps unsurprisingly, we and our coauthors had a range of views on how these results should be interpreted.

We have asked three of our coauthors to comment on what struck them most.

Peter Vesk, who was the source of the Eucalyptus data, said:

Looking at the mean of all the analyses, it makes sense. Grass has essentially a negligible effect on [the number of] eucalypt tree seedlings, compared to the distance from the nearest mother tree. But the range of estimated effects is gobsmacking. It fits with my own experience that lots of small differences in the analysis workflow can add to large variation [in results].

Simon Griffith collected the blue tit data more than 20 years ago, and it was not previously analysed due to the complexity of decisions about the right analytical pathway. He said:

This study demonstrates that there isn’t one answer from any set of data. There are a wide range of different outcomes and understanding the underlying biology needs to account for that diversity.

Meta-researcher Fiona Fidler, who studies research itself, said:

The point of these studies isn’t to scare people or to create a crisis. It is to help build our understanding of heterogeneity and what it means for the practice of science. Through metaresearch projects like this we can develop better intuitions about uncertainty and make better calibrated conclusions from our research.

What should we do about it?

In our view, the results suggest three courses of action for researchers, publishers, funders and the broader science community.

First, we should avoid treating published research as fact. A single scientific article is just one piece of evidence, existing in a broader context of limitations and biases.

The push for “novel” science means studying something that has already been investigated is discouraged, and consequently we inflate the value of individual studies. We need to take a step back and consider each article in context, rather than treating them as the final word on the matter.

Read more: The science 'reproducibility crisis' – and what can be done about it[13]

Second, we should conduct more analyses per article and report all of them. If research depends on what analytic choices are made, it makes sense to present multiple analyses to build a fuller picture of the result.

And third, each study should include a description of how the results depend on data analysis decision. Research publications tend to focus on discussing the ecological implications of their findings, but they should also talk about how different analysis choices influenced the results, and what that means for interpreting the findings.

Read more https://theconversation.com/two-questions-hundreds-of-scientists-no-easy-answers-how-small-differences-in-data-analysis-make-huge-differences-in-results-216177

The Times Features

Understanding the Dangers of Ignoring a Gas Leak

Gas leaks are silent threats lurking within both homes and workplaces. A gas leak occurs when natural gas or any other gaseous substance escapes from a pipeline or containment. T...

Can You Sell Your House Privately in Queensland? Here’s How

Selling a house privately in Queensland is entirely possible and can be a cost-effective alternative to using a real estate agent. While agents provide valuable expertise, their co...

Itinerary to Maximize Your Two-Week Adventure in Vietnam and Cambodia

Two weeks may not seem like much, but it’s just the right time for travelers to explore the best of Vietnam and Cambodia. From the bustling streets of Hanoi to the magnificent te...

How to Protect Your Garden Trees from Wind Damage in Australia

In Australia's expansive landscape, garden trees hold noteworthy significance. They not only enhance the aesthetic appeal of our homes but also play an integral role in the local...

Brisbane Homeowners Warned: Non-Compliant Flexible Hoses Pose High Flood Risk

As a homeowner in Brisbane, when you think of the potential for flood damage to your home, you probably think of weather events. But you should know that there may be a tickin...

Argan Oil-Infused Moroccanoil Shampoo: Nourish and Revitalize Your Hair

Are you ready to transform your hair from dull and lifeless to vibrant and full of life? Look no further than the luxurious embrace of Argan Oil-Infused Moroccanoil Shampoo! In a...

Times Magazine

"Eternal Nurture" by Cara Barilla: A Timeless Collection of Wisdom and Healing

Renowned Sydney-born author and educator Cara Barilla has released her latest book, Eternal Nurture, a profound collection of inspirational quotes designed to support mindfulness, emotional healing, and personal growth. With a deep commitment to ...

How AI-Driven SEO Enhancements Can Improve Headless CMS Content Visibility

Whereas SEO (search engine optimization) is critical in the digital landscape for making connections to content, much of it is still done manually keyword research, metatags, final tweaks at publication requiring a human element that takes extensiv...

Crypto Expert John Fenga Reveals How Blockchain is Revolutionising Charity

One of the most persistent challenges in the charity sector is trust. Donors often wonder whether their contributions are being used effectively or if overhead costs consume a significant portion. Traditional fundraising methods can be opaque, with...

Navigating Parenting Arrangements in Australia: A Legal Guide for Parents

Understanding Parenting Arrangements in Australia. Child custody disputes are often one of the most emotionally charged aspects of separation or divorce. Parents naturally want what is best for their children, but the legal process of determining ...

Blocky Adventures: A Minecraft Movie Celebration for Your Wrist

The Minecraft movie is almost here—and it’s time to get excited! With the film set to hit theaters on April 4, 2025, fans have a brand-new reason to celebrate. To honor the upcoming blockbuster, watchfaces.co has released a special Minecraft-inspir...

The Ultimate Guide to Apple Watch Faces & Trending Wallpapers

In today’s digital world, personalization is everything. Your smartwatch isn’t just a timepiece—it’s an extension of your style. Thanks to innovative third-party developers, customizing your Apple Watch has reached new heights with stunning designs...

LayBy Shopping