The Times Australia
The Times World News

.

Tests that diagnose diseases are less reliable than you’d expect. Here’s why

  • Written by Adrian Barnett, Professor of Statistics, Queensland University of Technology
Tests that diagnose diseases are less reliable than you’d expect. Here’s why

You feel unwell, and visit your doctor. They ask some questions and take some blood for testing; a few days later they call to say you have been diagnosed with a disease.

What are the chances you actually have the disease? For some common diagnostic tests, the answer is surprisingly low.

Few medical tests are 100% accurate. Part of the reason is that people are inherently variable, but many tests are also built on limited or biased samples of patients – and our own work has shown researchers may deliberately exaggerate[1] the effectiveness of new tests.

None of this means we should stop trusting diagnostic tests, but a better understanding of their strengths and weaknesses is essential if we want to use them wisely.

People are variable

An example of a widely used imperfect test is prostate-specific antigen (PSA) screening, which measures the level of a particular protein in the blood as an indicator of prostate cancer.

The test catches an estimated 93% of cancers – but it has a very high false positive rate, as around 80% of men with a positive result do not actually have cancer. For those in the 80%, the result creates unnecessary stress[2] and likely further testing including painful biopsies.

Read more: Prostate cancer testing: has the bubble burst?[3]

Rapid antigen tests for COVID-19 are another widely used imperfect test. A review of these tests[4] found that, of people without symptoms but with a positive test result, only 52% actually had COVID.

Among people with COVID symptoms and a positive result, the accuracy of the tests rose to 89%. This shows how a test’s performance cannot be summarised by a single number and depends on individual context.

Why aren’t diagnostic tests perfect? One key reason is that people are variable. A high temperature for you, for example, might be perfectly normal for someone else. For blood tests, many extraneous factors can influence the results, such as the time of day or how recently you have eaten.

Even the ubiquitous blood pressure test can be inaccurate[5]. Results can vary depending on whether the cuff is a good fit for your arm, if you have your legs crossed, and if you’re talking when the test is done.

Small samples and statistical skullduggery

There’s an enormous amount of research on new diagnostic models. New models frequently make the headlines as “medical breakthroughs”, such as how your handwriting could detect Parkinson’s disease[6], how your pharmacy loyalty card could detect ovarian cancer earlier[7], or how eye movements could detect schizophrenia[8].

But living up to the headlines is often a different story.

Many diagnostic models are developed based on small sample sizes. A review[9] found half of diagnostic studies used just over 100 patients. It is hard to get a true picture of the accuracy of a diagnostic test from such small samples.

For accurate results, the patients who use the test should be similar to those who were used to develop the test. For example, the widely used Framingham Risk Score for identifying people at high risk of heart disease was developed in the United States and is known to perform poorly[10] in Aboriginal and Torres Strait Islander people.

Similar disparities in accuracy have been found for “polygenic risk scores”. These combine information on thousands of genes to predict disease risk, but were developed in European populations and perform poorly in non-European populations[11].

Recently, we identified another important problem: researchers have exaggerated the accuracy of some models[12] to gain journal publications.

There are many ways to exaggerate the performance of a test, such as dropping hard-to-predict patients from the sample. Some tests are also not truly predictive, as they include information from the future, such as a predictive model of infection[13] that includes whether the patient had been prescribed antibiotics.

Read more: Elizabeth Holmes: Theranos scandal has more to it than just toxic Silicon Valley culture[14]

Perhaps the most extreme example of exaggerating the power of a diagnostic test was the Theranos scandal[15], in which a finger-prick blood test supposed to diagnose multiple health conditions attracted hundreds of millions of dollars from investors. This was too good to be true – and the mastermind has now been convicted of fraud.

Big data can’t make tests perfect

In the era of precision medicine and big data, it seems appealing to combine tens or hundreds of pieces of information about a patient – perhaps using machine learning or artificial intelligence – to provide highly accurate predictions. However, the promise is so far outstripping the reality.

One study[16] estimated 80,000 new prediction models were published between 1995 and 2020. That’s around 250 new models every month.

Are these models transforming healthcare? We see no sign of it – and if they really were having a big impact, surely we wouldn’t need such a steady stream of new models.

For many diseases there are data problems that no amount of sophisticated modelling can fix, such as measurement errors or missing data that make accurate predictions impossible.

Some diseases or illnesses are likely inherently random, and involve complex chains of events which a patient cannot describe and no model could predict. Examples might include injuries or previous illnesses that happened to a patient decades ago, which they cannot recall and are not in their medical notes.

Diagnostic tests will never be perfect. Acknowledging their imperfections will enable doctors and their patients to have an informed discussion about what a result means – and most importantly, what to do next.

References

  1. ^ deliberately exaggerate (bmcmedicine.biomedcentral.com)
  2. ^ creates unnecessary stress (theconversation.com)
  3. ^ Prostate cancer testing: has the bubble burst? (theconversation.com)
  4. ^ review of these tests (www.cochrane.org)
  5. ^ can be inaccurate (www.ama-assn.org)
  6. ^ handwriting could detect Parkinson’s disease (www.jpost.com)
  7. ^ detect ovarian cancer earlier (www.theguardian.com)
  8. ^ eye movements could detect schizophrenia (www.abdn.ac.uk)
  9. ^ A review (www.bmj.com)
  10. ^ perform poorly (pubmed.ncbi.nlm.nih.gov)
  11. ^ perform poorly in non-European populations (www.nature.com)
  12. ^ the accuracy of some models (bmcmedicine.biomedcentral.com)
  13. ^ predictive model of infection (www.statnews.com)
  14. ^ Elizabeth Holmes: Theranos scandal has more to it than just toxic Silicon Valley culture (theconversation.com)
  15. ^ Theranos scandal (theconversation.com)
  16. ^ study (osf.io)

Read more https://theconversation.com/tests-that-diagnose-diseases-are-less-reliable-than-youd-expect-heres-why-213359

Times Magazine

Headless CMS in Digital Twins and 3D Product Experiences

Image by freepik As the metaverse becomes more advanced and accessible, it's clear that multiple sectors will use digital twins and 3D product experiences to visualize, connect, and streamline efforts better. A digital twin is a virtual replica of ...

The Decline of Hyper-Casual: How Mid-Core Mobile Games Took Over in 2025

In recent years, the mobile gaming landscape has undergone a significant transformation, with mid-core mobile games emerging as the dominant force in app stores by 2025. This shift is underpinned by changing user habits and evolving monetization tr...

Understanding ITIL 4 and PRINCE2 Project Management Synergy

Key Highlights ITIL 4 focuses on IT service management, emphasising continual improvement and value creation through modern digital transformation approaches. PRINCE2 project management supports systematic planning and execution of projects wit...

What AI Adoption Means for the Future of Workplace Risk Management

Image by freepik As industrial operations become more complex and fast-paced, the risks faced by workers and employers alike continue to grow. Traditional safety models—reliant on manual oversight, reactive investigations, and standardised checklist...

From Beach Bops to Alpine Anthems: Your Sonos Survival Guide for a Long Weekend Escape

Alright, fellow adventurers and relaxation enthusiasts! So, you've packed your bags, charged your devices, and mentally prepared for that glorious King's Birthday long weekend. But hold on, are you really ready? Because a true long weekend warrior kn...

Effective Commercial Pest Control Solutions for a Safer Workplace

Keeping a workplace clean, safe, and free from pests is essential for maintaining productivity, protecting employee health, and upholding a company's reputation. Pests pose health risks, can cause structural damage, and can lead to serious legal an...

The Times Features

Duke of Dural to Get Rooftop Bar as New Owners Invest in Venue Upgrade

The Duke of Dural, in Sydney’s north-west, is set for a major uplift under new ownership, following its acquisition by hospitality group Good Beer Company this week. Led by resp...

Prefab’s Second Life: Why Australia’s Backyard Boom Needs a Circular Makeover

The humble granny flat is being reimagined not just as a fix for housing shortages, but as a cornerstone of circular, factory-built architecture. But are our systems ready to s...

Melbourne’s Burglary Boom: Break-Ins Surge Nearly 25%

Victorian homeowners are being warned to act now, as rising break-ins and falling arrest rates paint a worrying picture for suburban safety. Melbourne residents are facing an ...

Exploring the Curriculum at a Modern Junior School in Melbourne

Key Highlights The curriculum at junior schools emphasises whole-person development, catering to children’s physical, emotional, and intellectual needs. It ensures early year...

Distressed by all the bad news? Here’s how to stay informed but still look after yourself

If you’re feeling like the news is particularly bad at the moment, you’re not alone. But many of us can’t look away – and don’t want to. Engaging with news can help us make ...

The Role of Your GP in Creating a Chronic Disease Management Plan That Works

Living with a long-term condition, whether that is diabetes, asthma, arthritis or heart disease, means making hundreds of small decisions every day. You plan your diet against m...