The Times Australia
The Times World News

.

why automated systems must put the human factor first

  • Written by Mark Rickerby, Lecturer, School of Product Design, University of Canterbury
why automated systems must put the human factor first

The incident of a woman misidentified by facial recognition technology[1] at a Rotorua supermarket should have come as no surprise.

When Foodstuffs North Island announced its intention to trial this technology in February, as part of a strategy to combat retail crime, technology and privacy experts immediately raised concerns.

In particular, the risk of Māori women and women of colour being discriminated against[2] was raised, and has now been borne out by what happened in early April to Te Ani Solomon.

Speaking to media this week, Solomon said[3] she thought ethnicity was a “huge factor” in her wrongful identification. “Unfortunately, it will be the experience of many Kiwis if we don’t have some rules and regulations around this.”

The supermarket company’s response[4] that this was a “genuine case of human error” fails to address the deeper questions about such use of AI and automated systems.

Automated decisions and human actions

Automated facial recognition is often discussed in the abstract – as pure algorithmic pattern matching, with emphasis on assessing correctness and accuracy.

These are rightfully important priorities for systems that deal with biometric data and security. But with such crucial focus on the results of automated decisions, it’s easy to overlook concerns about how these decisions are applied.

Designers use the term “context of use” to describe the everyday working conditions, tasks and goals of a product. With facial recognition technology in supermarkets, the context of use goes far beyond traditional design concerns such as ergonomics or usability.

Read more: The use of technology in policing should be regulated to protect people from wrongful convictions[5]

It requires consideration of how automated trespass notifications trigger in-store responses, protocols for managing those responses, and what happens when things go wrong. These are more than just pure technology or data problems.

This perspective helps us understand and balance the impact of engineering and design interventions at different levels of a system.

Investing in improving prediction accuracy seems an obvious priority for facial recognition systems. But this has to be seen in a broader context of use where the harm done by a small number of wrong predictions outweighs marginal performance improvements elsewhere.

Responding to retail crime

New Zealand is not alone in reported increases in shoplifting and violent behaviour in stores. In the UK, it has been described as a “crisis”, with assaulting a retail worker now a standalone criminal offence[6].

Canadian police are funnelling extra resources[7] into “shoplifting crackdowns”. And in California, retail giants Walmart and Target are pushing for increased penalties[8] for retail crime.

While these problems have been linked to the rising cost of living, industry group Retail NZ has pointed to[9] profit-seeking organised crime as the major factor.

Read more: Facial recognition technology could soon be everywhere – here's how to make it safer[10]

Sensationalised coverage using security footage of brazen thefts and assaults in stores is undoubtedly influencing public perception. But a trend is difficult to measure due to a lack of consistent, impartial data on shoplifting and offenders.

It is estimated that 15-20% of people in New Zealand are affected by food insecurity, a problem found to be[11] strongly associated with ethnicity and socioeconomic position. The links between cost of living, food insecurity and black market distribution of stolen groceries are likely to be complex and nuanced.

Caution is therefore needed when assessing cause and effect, given the risks of harm and implications for civil society of a shift towards constant surveillance in retail spaces.

AI technologies need ‘humans in the loop’ to avoid bias and error. Getty Images

AI and human bias

Commendably, Foodstuffs has engaged with the Privacy Commissioner, and has been transparent about safeguards[12] in biometric data collection and deletion protocols. What’s missing is more clarity around protocols for the security response in stores.

This is more than about customers consenting to facial recognition cameras. Customers also need to know what happens when a trespass notification is issued, and the dispute resolution process should a misidentification occur.

Read more: Avoiding a surveillance society: how better rules can rein in facial recognition tech[13]

Research suggests human decision makers can inherit biases from AI decisions[14]. In situations of heightened stress and risk of violence, combining automated facial recognition with ad-hoc human judgement is potentially dangerous.

Rather than isolating and blaming individual workers or technology components as single points of failure, there needs to be more emphasis on resilience and tolerance for error across the whole system.

AI errors and human errors cannot be avoided entirely. AI security protocols with “humans in the loop” need more careful safeguards that respect customer rights and protect against stereotyping.

Read more: The secret sauce of Coles’ and Woolworths’ profits: high-tech surveillance and control[15]

Shopping and surveillance

Australian supermarkets have responded to retail crime with overt technological surveillance[16]: body cameras issued to staff (also now adopted[17] by Woolworths in New Zealand), digitally tracking customer movement through stores, automated trolley locks and exit gates to prevent people leaving without paying.

Excerpt from a 1979 IBM training manual. MIT-CSAIL[18]

Supermarkets may now be at the forefront of a technological shift in the shopping experience. Moving towards a surveillance culture where every customer is monitored as a potential thief is reminiscent of the ways global airport security changed after 9/11.

New Zealand product designers, software engineers and data scientists will be paying close attention to the outcome of the Privacy Commissioner’s review of the Foodstuffs facial recognition trial[19].

Theft and violence is an urgent problem for supermarkets to address. But they now need to show that digital surveillance systems are a more responsible, ethical and effective solution than possible alternative approaches.

This means acknowledging technology requires human-centered design to avoid misuse, bias and harm. In turn, this can help guide regulatory frameworks and standards, inform public debate on the acceptable use of AI, and support development of safer automated systems.

References

  1. ^ misidentified by facial recognition technology (www.nzherald.co.nz)
  2. ^ being discriminated against (www.teaonews.co.nz)
  3. ^ Solomon said (www.1news.co.nz)
  4. ^ response (taiuru.co.nz)
  5. ^ The use of technology in policing should be regulated to protect people from wrongful convictions (theconversation.com)
  6. ^ standalone criminal offence (www.gov.uk)
  7. ^ funnelling extra resources (www.mapleridgenews.com)
  8. ^ pushing for increased penalties (www.bnnbloomberg.ca)
  9. ^ Retail NZ has pointed to (www.stuff.co.nz)
  10. ^ Facial recognition technology could soon be everywhere – here's how to make it safer (theconversation.com)
  11. ^ found to be (www.growingup.co.nz)
  12. ^ transparent about safeguards (www.nzherald.co.nz)
  13. ^ Avoiding a surveillance society: how better rules can rein in facial recognition tech (theconversation.com)
  14. ^ inherit biases from AI decisions (www.nature.com)
  15. ^ The secret sauce of Coles’ and Woolworths’ profits: high-tech surveillance and control (theconversation.com)
  16. ^ overt technological surveillance (theconversation.com)
  17. ^ now adopted (www.rnz.co.nz)
  18. ^ MIT-CSAIL (www.csail.mit.edu)
  19. ^ review of the Foodstuffs facial recognition trial (www.privacy.org.nz)

Read more https://theconversation.com/supermarket-facial-recognition-failure-why-automated-systems-must-put-the-human-factor-first-228284

Times Magazine

Building a Strong Online Presence with Katoomba Web Design

Katoomba web design is more than just creating a website that looks good—it’s about building an online presence that reflects your brand, engages your audience, and drives results. For local businesses in the Blue Mountains, a well-designed website a...

September Sunset Polo

International Polo Tour To Bridge Historic Sport, Life-Changing Philanthropy, and Breath-Taking Beauty On Saturday, September 6th, history will be made as the International Polo Tour (IPT), a sports leader headquartered here in South Florida...

5 Ways Microsoft Fabric Simplifies Your Data Analytics Workflow

In today's data-driven world, businesses are constantly seeking ways to streamline their data analytics processes. The sheer volume and complexity of data can be overwhelming, often leading to bottlenecks and inefficiencies. Enter the innovative da...

7 Questions to Ask Before You Sign IT Support Companies in Sydney

Choosing an IT partner can feel like buying an insurance policy you hope you never need. The right choice keeps your team productive, your data safe, and your budget predictable. The wrong choice shows up as slow tickets, surprise bills, and risky sh...

Choosing the Right Legal Aid Lawyer in Sutherland Shire: Key Considerations

Legal aid services play an essential role in ensuring access to justice for all. For people in the Sutherland Shire who may not have the financial means to pay for private legal assistance, legal aid ensures that everyone has access to representa...

Watercolor vs. Oil vs. Digital: Which Medium Fits Your Pet's Personality?

When it comes to immortalizing your pet’s unique personality in art, choosing the right medium is essential. Each artistic medium, whether watercolor, oil, or digital, has distinct qualities that can bring out the spirit of your furry friend in dif...

The Times Features

NSW has a new fashion sector strategy – but a sustainable industry needs a federally legislated response

The New South Wales government recently announced the launch of the NSW Fashion Sector Strategy, 2025–28[1]. The strategy, developed in partnership with the Australian Fashion ...

From Garden to Gift: Why Roses Make the Perfect Present

Think back to the last time you gave or received flowers. Chances are, roses were part of the bunch, or maybe they were the whole bunch.   Roses tend to leave an impression. Even ...

Do I have insomnia? 5 reasons why you might not

Even a single night of sleep trouble can feel distressing and lonely. You toss and turn, stare at the ceiling, and wonder how you’ll cope tomorrow. No wonder many people star...

Wedding Photography Trends You Need to Know (Before You Regret Your Album)

Your wedding album should be a timeless keepsake, not something you cringe at years later. Trends may come and go, but choosing the right wedding photography approach ensures your ...

Can you say no to your doctor using an AI scribe?

Doctors’ offices were once private. But increasingly, artificial intelligence (AI) scribes (also known as digital scribes) are listening in. These tools can record and trans...

There’s a new vaccine for pneumococcal disease in Australia. Here’s what to know

The Australian government announced last week there’s a new vaccine[1] for pneumococcal disease on the National Immunisation Program for all children. This vaccine replaces pr...