Prefer WhistleOut in Google
During a Senate inquiry into Triple Zero service outages, Telstra CEO Vicki Brady said no telco can guarantee that outages won't happen.
"Networks are certainly not infallible. Our job is to make sure we're taking the steps that will mitigate as best we can," said Brady.
"I would love to be able to sit here and say there will be zero outages; [but] the reality of a complex network environment with fast-evolving technology, no telco around the world could guarantee that.
In addition, Brady revealed some new information about the cause of the outage, but made it clear an external investigation is still underway to confirm Telstra’s findings.
We’ll go over the broad strokes of her statement, including the newer information, and provide some additional context around the Telstra outage.
The outage occurred last Wednesday morning, first detected by Telstra at 4:20am. The telco posted an update on its website 18 minutes later, confirming the issue was under investigation. Brady has now confirmed the telco believes the inciting event was at 3:38am when Telstra staff performed maintenance on a Network Time Protocol server, prompted by a backup power fault.
The maintenance team reportedly followed the correct procedures, but when the server restarted it did so with the wrong date. As implied by the name, the Network Time Protocol server is partly responsible for making sure different parts of the Telstra network keep accurate time with each other. Thanks to the restart error, an incorrect date and time began to spread through the network, putting the network out of sync and causing voice and data connections to fail.
At its peak, the outage affected around 45% of all voice and data sessions on Telstra’s network.
Brady reports the device was isolated at 7:11am, after which Telstra could begin restoring the network. By 10am, 90% of the network had been restored and Telstra stated it had conducted successful Triple Zero checks.
At a conference in the early evening, Telstra CFO Michael Ackland reported the outage had been fully resolved by 4pm and re-confirmed the telco’s belief that Triple Zero was functioning successfully. But he also discussed 333 failed or dropped emergency calls had been detected. This was a larger volume than expected and, at the time, Ackland surmised it could be due to people testing if they could connect to Triple Zero, then hanging up once the call was successful.
That night, Telstra discovered the initial fault had caused an additional issue (requiring a different fix), which was causing some Triple Zero calls to fail. The following morning, Telstra reported its Triple Zero outage had been fixed.
In her statement today, Brady said a total of 604 Triple Zero calls failed due to this second error. She went on to state Telstra successfully completed 604 welfare checks, 144 of which had to be escalated to in-person checks performed by police.
At this time, Brady is unaware of any life-threatening or fatal incidents caused by this latest Triple Zero outage – something confirmed by Communications minister Anika Wells on Friday last week.
What caused the error?
Telstra believes the reason its Network Time Protocol server restarted with the wrong time and date was due to an “intentional design change” to a GPS card that had not been properly documented. There was also a software update that had not been applied to the card.
Brady claims if either of these oversights hadn’t happened, the outage might have been avoided – which she described as “clearly unacceptable”.
As for why this design change wasn’t properly documented and why the software update had not been installed, Telstra is waiting on the outcome of its external investigation to explain this.
What happens next?
We're waiting for Telstra’s external investigation to wrap up, so the telco itself hasn't announced any changes to its standard procedures.
But Telstra’s CEO confirmed the telco will communicate transparently about its investigation's outcomes.
“I want to be clear with the Committee and with Australians: we will be open about what we find through our investigation, cooperate fully with this inquiry and with regulators, and take the actions required to strengthen the resilience of our network and the services Australians rely on.”
Brady also added another apology to the several Telstra has made over the last week.
"Last week, Telstra let Australians down. We let our customers down, we let the community down… For this I am deeply sorry."
Alex Angove-Plumb
Digital Content Editor
Related Articles
