Sat Feb 5

The COVIDSafe app was designed to help contact tracers. We crunched the numbers to see what really happened

Written by: Florian Vogt, Senior Research Fellow, The Kirby Institute, UNSW

Early in the pandemic, the COVIDSafe app ^[1] was designed as a tool to improve contact tracing. And a multimedia campaign promoted it as helping to find more COVID contacts faster.

Now, almost two years after its launch in April 2020, we publish in The Lancet Public Health ^[2] our evaluation of the app’s effectiveness and usefulness in New South Wales.

We analysed data from all 619 COVID cases recorded in NSW above the age of 12 and their contacts between May and November 2020. We also interviewed contact tracers about whether they considered the app useful.

Our analysis showed the app did not add much value to the existing, conventional contact tracing system.

A total of 22% of cases were using the app. Most (61%) contacts the app registered as “close contacts” turned out not to be epidemiologically linked to a case. The app detected only 15% of true close contacts identified by conventional contact tracing.

In total, COVIDSafe detected only 17 additional true close contacts in NSW during the six-month evaluation period.

It caused substantial additional work for contact tracers and overall, did not make a meaningful contribution to the COVID response in NSW.

Tracing takes time

Interviewing new cases to identify, trace and isolate their close contacts is a crucial public health activity. But this is time-consuming and contact tracers can become rapidly overwhelmed when case numbers rise.

Digital contact tracing apps were enthusiastically embraced early in the COVID pandemic in many countries ^[4], including Australia.

Like many other tracing apps, COVIDSafe uses smartphones’ built-in Bluetooth function to exchange signals ^[5] between phones.

The duration, frequency and transmission strength of these “digital handshakes” are used to determine whether two smartphone users have come into “close contact” with each other.

Low ability to detect and identify close contacts correctly

During our study period, less than a quarter of all cases were using the app. This is half the proportion of the Australian general public ^[6] who did so.

Of those cases who used the app, many had not a single contact registered by the app. For others, contact tracers could not access the app data.

Contact tracers could use app data for 32 (5%) of the 619 cases over our study period. Of those cases, only 79 (39%) of the 205 contacts the app registered as “close contacts” could be verified as true close contacts. This suggests poor positive predictive value of the app.

Examples of false close contacts the app registered were:

neighbours in different flats in apartment buildings
office workers in adjacent rooms
customers in neighbouring restaurants
people waiting in separate cars at COVID drive-through testing clinics.

The vast majority (85%) of close contacts identified by conventional contact tracing were not detected by the app, indicating low sensitivity.

A few additional contacts detected

During the six months of our study, there were only 17 true close contacts identified by COVIDSafe who would have otherwise been missed – a tiny fraction of the more than 25,300 close contacts detected and followed up through conventional contact tracing in NSW during the same period.

None of these 17 contacts became positive. So COVIDSafe did not contribute to preventing any new exposures in NSW during our evaluation period.

Technological flaws

Contact tracers did not find the app easy to use. Some said the app did not seem to work as reliably on all types of phones. The number of contacts on iPhones were substantially underestimated, while those from Android phones were overestimated.

Contact tracers also noted the app’s apparent inability to properly register contacts unless it was open. This might explain the large number of cases without any registered contacts in our study.

The process of cross-matching close contacts identified by the app with those identified through case interviews was seen as time-consuming, particularly as most contacts picked up by the app were not really close contacts.

Interviewed staff said this could easily overwhelm the contact tracing system had case numbers been higher, paradoxically leading to a reduction in usefulness of the app when it would be most needed.

Overall, contact tracers’ perceptions of the app ranged from “not impacting much” to being an additional step that increased workload without adding much value.

smart phone with covidsafe app info displayed

The app identified 17 contacts that would have been otherwise missed. AAP Image/Scott Barbour ^[8]

Little added value at high costs

In our study, COVIDSafe did not make a meaningful contribution to the COVID response in NSW during 2020. Instead, the app created a high workload for no clear benefit.

This stands against the A$7.7 million it has cost ^[9] to develop and run COVIDSafe until the end of April 2021, with an estimated $60,000-$75,000 per month in maintenance since.

The arrival of the highly transmissible Omicron variant ^[10] in late 2021 in Australia, coinciding with the lifting of most public health restrictions, led to massive expansion in case numbers, forcing a series of major adjustments to contact tracing.

In most parts of Australia, contact tracing now has a very limited role in COVID control. Nevertheless, it is likely to remain ^[12] a key public health intervention for infectious diseases in the future.

For digital contact tracing apps to be effective and useful, it will be important to involve contact tracers when designing the system, to road test the underlying technology in real-life settings, and to evaluate the app regularly after roll-out.

Dr Anthea Katelaris, a public health physician who worked at the Western Sydney Local Health District public health unit at the time of the study, co-authored the research mentioned in this article.