Tue Nov 14

Optus has revealed the cause of the major outage. Could it happen again?

Written by Mark A Gregory, Associate Professor, School of Engineering, RMIT University

Around 4.05am on Wednesday November 8 2023, Optus suffered a nationwide network outage lasting well into the evening, more than 12 hours later.

Now, Optus has released some information on what happened, stating ^[1] “we now know what the cause was and have taken steps to ensure it will not happen again”.

As a telecommunications expert, I believe we should have no confidence in this statement, because the poorly worded explanation leaves many questions unanswered.

Could a similar outage happen again? We don’t know – but there are ways to make it less likely.

How did the outage unfold?

The Optus outage caused all services to go offline. Landlines, mobile phones, home internet, small business and enterprise, and cloud connections all dropped out.

The most serious impact of the outage was that Optus landlines couldn’t dial 000 ^[3] and Optus mobile phones were unable to connect to the 000 emergency call service unless the connection occurred through Telstra or Vodafone infrastructure.

More than 10 million Optus customers were affected by the outage that brought Melbourne’s trains to a halt ^[4] and left Optus’s small business customers unable to carry out EFTPOS transactions ^[5].

So, what went wrong with Optus?

Optus has revealed that a “routine software upgrade” triggered a cascading failure in the Optus internet protocol ^[6] (IP) core network – the central backbone of their network that authorises device access and provides customer management.

Optus has provided a brief answer on why the entire network went offline ^[7]:

“These routing information changes propagated through multiple layers in our network and exceeded preset safety levels on key routers which could not handle these. This resulted in those routers disconnecting from the Optus IP Core network to protect themselves.”

Routing information is used to find a path from one location on the internet to another – a router is a device that manages the traffic flows.

The explanation provided by Optus points to human error. This confirms what industry experts suspected had happened ^[8]. The resulting flood of “routing information changes” overwhelmed ^[9] key routers in the core network causing them to disconnect, thereby bringing the entire network to a halt.

Photo of a white wifi router on a desk with a person working on laptop in the backround

Your internet router is a home version of a device that manages data traffic flow. Teerasan Phutthigorn/Shutterstock ^[11]

Should the outage have been preventable?

Outages of this kind are not uncommon – human error has led to major companies going offline in the past.

But an entire telecommunications network going offline is unusual. The network should be designed in such a way that redundancy (backups) and resiliency are built in from the outset.

Before a software upgrade occurs, there should be modelling, testing and several layers of sign-off.

In case something goes wrong, there should be infrastructure and system redundancy. An automated or manual procedure should exist to ensure the redundant systems become operational within a few minutes.

In 2021, Facebook, WhatsApp and Instagram ^[12] disappeared from the internet for roughly six hours due to an incorrect routing configuration.

Meta’s lengthy and informative statement ^[13] at the time provides an example of the level of detail that we should expect Optus to provide.

With the Optus outage and similar incidents at other companies that have led to major outages, in nearly every case the outage was preventable and highlighted deficiencies in the organisation.

What should Optus do now?

The national outage means the Optus network is not fit for purpose ^[15].

It can be assumed Optus has a number of deficiencies, such as problems with engineering capability, testing, procedures, network redundancy and resilience.

Optus states they are “committed to learning from what has occurred” and will continue to work to “increase the resilience” of their network.

For this to lead to an effective outcome, Optus will need to carry out a review and put in place new processes, infrastructure and systems to prevent a similar outage in the future.

How do we know a similar outage won’t happen again?

We don’t.

We need enhanced government regulation of the Australian telecommunications network operators to provide improved visibility of the redundancy and resilience of their networks. The Senate has commenced an inquiry ^[16] into the Optus outage.

Telecommunications is an essential service. Australians should be able to connect to the 000 emergency call service at all times. Reliable access to medical services, EFTPOS and the internet are vital.

If necessary, penalties should be introduced into the Telecommunications Act 1997 ^[17] to ensure telecommunications network operators implement and maintain “best practice” related to network operation, redundancy and resilience.