At a two-and-a-half-hour Senate grilling on Friday, Telstra executives revealed just how bad their catastrophic outage earlier this month was. At its peak, it affected about 45% of calls and data across the telco giant’s national network.
The root cause was a failed software update of a piece of hardware – specifically, a server, manufactured in 2011 and worth roughly $30,000, that helps keep accurate time across the network.
Telstra noticed some issues with the time server in October last year. It had received warnings from the manufacturer of the server in 2022 and again in January this year. The telco did not act on these warnings. Under repeated questioning, Gerard Tracy, Telstra’s executive for end-to-end resilience, conceded that replacing the server could have avoided the outage.
But Telstra isn’t alone in its neglect of key physical infrastructure that underpins its systems. In fact, this problem is widespread across many industries. And this outage serves as a major wake-up call.
Telstra forgot the time
Fundamentally, Telstra forgot the time. Digital devices tell the time thanks to systems known as time servers , which prevent clocks from drifting out of alignment. To maintain accuracy, these servers rely on signals from GPS satellites.
However, GPS satellites don’t broadcast the date as we read it. Rather, they broadcast the number of weeks since the network launched. Due to a programming quirk, this counter resets every 1,024 weeks (roughly 19 and a half years).
This tiny quirk triggered Telstra’s collapse. After a routine restart, the ageing Telstra time server reconnected to the GPS network, but misinterpreted the 20-year reset cycle.
It thought the year was 2006 , not 2026. And because Telstra operated too few redundant time servers, its broader network couldn’t achieve a consensus on what the true time actually was, causing a cascading, system-wide failure.
An issue far beyond Telstra
This GPS quirk has repeatedly caught out major brands, including Honda , Porsche and GoPro .
The Department of Home Affairs’ Cyber and Infrastructure Security Centre issued a nationwide warning in 2024 about how timekeeping systems serve as a key strategic vunerability .
But our failure to maintain systems extends much beyond clocks. And when these systems fail, the ripple effects can be huge.
Recent history is littered with such failures.
In 2024, devices across the whole economy – from ATMs to web servers – went down due to a failed software update from the internet infrastructure company Cloudstrike.
Similarly, in 2023, Optus had a single device fail during a routine software update, which caused cascading issues across the whole economy.
In the United States, Southwest Airlines had to cancel more than 16,000 flights in 2022 because ancient flight-planning software couldn’t handle changes in the weather .
In each of these cases, the financial toll was felt not just by the business experiencing the failure, but also by the broader economy.
It’s time for change
Without change, these kinds of incidents will continue. However, making such changes is easier said than done.
Southwest Airlines forecast that it wouldn’t be able to fix the flight-planning system that failed in 2022 until 2028 .
The financial sector is particularly vulnerable to these issues. Banking systems are built on a software platform called COBOL, an antiquated language that only a small (and dwindling) handful of programmers can use .
Despite underpinning 80% of credit-card transactions, 95% of ATM transactions and 40% of online banking, and processing trillions of dollars in transactions every day , financial institutions have repeatedly shelved modernisation efforts for being too difficult.
Beyond the risks associated with them, the costs of running these legacy systems can be huge.
In the financial sector it’s estimated 75% of computing budgets are spent on legacy systems .
Why do these issues persist?
One factor in our reliance on legacy systems is how modernisation efforts are accounted for in corporate balance sheets.
Spending $30,000 on a new time server, or millions of dollars on a software modernisation effort, will typically be treated as a operational expense rather than a capital investment.
Unlike artificial intelligence projects or other flashy IT spends, the absence of a disaster produces no revenue, making preventative maintenance and risk reduction invisible on quarterly earnings reports.
Such a shortsighted view on shareholder value maximisation will now potentially cost Telstra more than A$30 million in fines , on top of compensation costs .
This is a lesson Telstra has learned before.
In 2012, a fire broke out at a Telstra facility in Warnambool , resulting in an estimated cost of more than $20 million. However, when the company rebuilt the facility, it found a significant amount of its old hardware was redundant, and decomissioning this across the country reduced its nationwide energy bill significantly .
As this shows, modernising systems can unlock genuine financial value – from efficiency dividends to minimised corporate risk. The challenge lies in articulating that long-term value to boards and shareholders focused on immediate returns.
Shoring up the load-bearing infrastructure that supports our digital world will not be easy or cheap. And while we can never fully eliminate every digital vulnerability , corporations and governments must do far more to aggressively manage our exposure.
