The Times Australia
The Times World News

.

Tests that diagnose diseases are less reliable than you’d expect. Here’s why

  • Written by Adrian Barnett, Professor of Statistics, Queensland University of Technology
Tests that diagnose diseases are less reliable than you’d expect. Here’s why

You feel unwell, and visit your doctor. They ask some questions and take some blood for testing; a few days later they call to say you have been diagnosed with a disease.

What are the chances you actually have the disease? For some common diagnostic tests, the answer is surprisingly low.

Few medical tests are 100% accurate. Part of the reason is that people are inherently variable, but many tests are also built on limited or biased samples of patients – and our own work has shown researchers may deliberately exaggerate[1] the effectiveness of new tests.

None of this means we should stop trusting diagnostic tests, but a better understanding of their strengths and weaknesses is essential if we want to use them wisely.

People are variable

An example of a widely used imperfect test is prostate-specific antigen (PSA) screening, which measures the level of a particular protein in the blood as an indicator of prostate cancer.

The test catches an estimated 93% of cancers – but it has a very high false positive rate, as around 80% of men with a positive result do not actually have cancer. For those in the 80%, the result creates unnecessary stress[2] and likely further testing including painful biopsies.

Read more: Prostate cancer testing: has the bubble burst?[3]

Rapid antigen tests for COVID-19 are another widely used imperfect test. A review of these tests[4] found that, of people without symptoms but with a positive test result, only 52% actually had COVID.

Among people with COVID symptoms and a positive result, the accuracy of the tests rose to 89%. This shows how a test’s performance cannot be summarised by a single number and depends on individual context.

Why aren’t diagnostic tests perfect? One key reason is that people are variable. A high temperature for you, for example, might be perfectly normal for someone else. For blood tests, many extraneous factors can influence the results, such as the time of day or how recently you have eaten.

Even the ubiquitous blood pressure test can be inaccurate[5]. Results can vary depending on whether the cuff is a good fit for your arm, if you have your legs crossed, and if you’re talking when the test is done.

Small samples and statistical skullduggery

There’s an enormous amount of research on new diagnostic models. New models frequently make the headlines as “medical breakthroughs”, such as how your handwriting could detect Parkinson’s disease[6], how your pharmacy loyalty card could detect ovarian cancer earlier[7], or how eye movements could detect schizophrenia[8].

But living up to the headlines is often a different story.

Many diagnostic models are developed based on small sample sizes. A review[9] found half of diagnostic studies used just over 100 patients. It is hard to get a true picture of the accuracy of a diagnostic test from such small samples.

For accurate results, the patients who use the test should be similar to those who were used to develop the test. For example, the widely used Framingham Risk Score for identifying people at high risk of heart disease was developed in the United States and is known to perform poorly[10] in Aboriginal and Torres Strait Islander people.

Similar disparities in accuracy have been found for “polygenic risk scores”. These combine information on thousands of genes to predict disease risk, but were developed in European populations and perform poorly in non-European populations[11].

Recently, we identified another important problem: researchers have exaggerated the accuracy of some models[12] to gain journal publications.

There are many ways to exaggerate the performance of a test, such as dropping hard-to-predict patients from the sample. Some tests are also not truly predictive, as they include information from the future, such as a predictive model of infection[13] that includes whether the patient had been prescribed antibiotics.

Read more: Elizabeth Holmes: Theranos scandal has more to it than just toxic Silicon Valley culture[14]

Perhaps the most extreme example of exaggerating the power of a diagnostic test was the Theranos scandal[15], in which a finger-prick blood test supposed to diagnose multiple health conditions attracted hundreds of millions of dollars from investors. This was too good to be true – and the mastermind has now been convicted of fraud.

Big data can’t make tests perfect

In the era of precision medicine and big data, it seems appealing to combine tens or hundreds of pieces of information about a patient – perhaps using machine learning or artificial intelligence – to provide highly accurate predictions. However, the promise is so far outstripping the reality.

One study[16] estimated 80,000 new prediction models were published between 1995 and 2020. That’s around 250 new models every month.

Are these models transforming healthcare? We see no sign of it – and if they really were having a big impact, surely we wouldn’t need such a steady stream of new models.

For many diseases there are data problems that no amount of sophisticated modelling can fix, such as measurement errors or missing data that make accurate predictions impossible.

Some diseases or illnesses are likely inherently random, and involve complex chains of events which a patient cannot describe and no model could predict. Examples might include injuries or previous illnesses that happened to a patient decades ago, which they cannot recall and are not in their medical notes.

Diagnostic tests will never be perfect. Acknowledging their imperfections will enable doctors and their patients to have an informed discussion about what a result means – and most importantly, what to do next.

References

  1. ^ deliberately exaggerate (bmcmedicine.biomedcentral.com)
  2. ^ creates unnecessary stress (theconversation.com)
  3. ^ Prostate cancer testing: has the bubble burst? (theconversation.com)
  4. ^ review of these tests (www.cochrane.org)
  5. ^ can be inaccurate (www.ama-assn.org)
  6. ^ handwriting could detect Parkinson’s disease (www.jpost.com)
  7. ^ detect ovarian cancer earlier (www.theguardian.com)
  8. ^ eye movements could detect schizophrenia (www.abdn.ac.uk)
  9. ^ A review (www.bmj.com)
  10. ^ perform poorly (pubmed.ncbi.nlm.nih.gov)
  11. ^ perform poorly in non-European populations (www.nature.com)
  12. ^ the accuracy of some models (bmcmedicine.biomedcentral.com)
  13. ^ predictive model of infection (www.statnews.com)
  14. ^ Elizabeth Holmes: Theranos scandal has more to it than just toxic Silicon Valley culture (theconversation.com)
  15. ^ Theranos scandal (theconversation.com)
  16. ^ study (osf.io)

Read more https://theconversation.com/tests-that-diagnose-diseases-are-less-reliable-than-youd-expect-heres-why-213359

Times Magazine

Building an AI-First Culture in Your Company

AI isn't just something to think about anymore - it's becoming part of how we live and work, whether we like it or not. At the office, it definitely helps us move faster. But here's the thing: just using tools like ChatGPT or plugging AI into your wo...

Data Management Isn't Just About Tech—Here’s Why It’s a Human Problem Too

Photo by Kevin Kuby Manuel O. Diaz Jr.We live in a world drowning in data. Every click, swipe, medical scan, and financial transaction generates information, so much that managing it all has become one of the biggest challenges of our digital age. Bu...

Headless CMS in Digital Twins and 3D Product Experiences

Image by freepik As the metaverse becomes more advanced and accessible, it's clear that multiple sectors will use digital twins and 3D product experiences to visualize, connect, and streamline efforts better. A digital twin is a virtual replica of ...

The Decline of Hyper-Casual: How Mid-Core Mobile Games Took Over in 2025

In recent years, the mobile gaming landscape has undergone a significant transformation, with mid-core mobile games emerging as the dominant force in app stores by 2025. This shift is underpinned by changing user habits and evolving monetization tr...

Understanding ITIL 4 and PRINCE2 Project Management Synergy

Key Highlights ITIL 4 focuses on IT service management, emphasising continual improvement and value creation through modern digital transformation approaches. PRINCE2 project management supports systematic planning and execution of projects wit...

What AI Adoption Means for the Future of Workplace Risk Management

Image by freepik As industrial operations become more complex and fast-paced, the risks faced by workers and employers alike continue to grow. Traditional safety models—reliant on manual oversight, reactive investigations, and standardised checklist...

The Times Features

Flipping vs. Holding: Which Investment Strategy Is Right for You?

Are you wondering whether flipping a property or holding onto it is the better investment strategy? The answer isn’t one-size-fits-all. Both strategies have distinct advantages a...

Why Everyone's Talking About Sea Moss - And Should You Try It Too?

Sea moss - a humble marine plant that’s been used for centuries - is making a major comeback in modern wellness circles. And it’s not just a trend. With growing interest from athle...

A Guide to Smarter Real Estate Accounting: What You Might Be Overlooking

Real estate accounting can be a complex terrain, even for experienced investors and property managers. From tracking rental income to managing property expenses, the financial in...

What Is the Dreamtime? Understanding Aboriginal Creation Stories Through Art

Aboriginal culture is built on the deep and important meaning of Dreamtime, which links beliefs and history with the elements that make life. It’s not just myths; the Dreamtime i...

How Short-Term Lenders Offer Long-Lasting Benefits in Australia

In the world of personal and business finance, short-term lenders are often viewed as temporary fixes—quick solutions for urgent cash needs. However, in Australia, short-term len...

Why School Breaks Are the Perfect Time to Build Real Game Skills

School holidays provide uninterrupted time to focus on individual skill development Players often return sharper and more confident after structured break-time training Holid...