Telstra Addresses Issues Behind Triple Zero Transfer Failures

The post-mortem analysis conducted by Telstra following the March 1 Triple Zero outage revealed a combination of technical, process, and communication failures. The outage, which lasted for 90 minutes, was partly attributed to unexpected software failures occurring while medical support devices were attempting to log into the Telstra network. As a result of the outage, emergency calls had to be manually transferred to emergency services, leading to disruptions in service delivery. Tragically, one Victorian man reportedly died of a cardiac arrest during the outage.

The post-mortem findings underscored the importance of addressing technical vulnerabilities, streamlining processes, and improving communication protocols to prevent similar incidents in the future. Telstra acknowledged the need for enhanced software reliability and robustness, as well as improved procedures for handling emergency situations.

Additionally, efforts to enhance coordination and communication with relevant stakeholders, including emergency service providers and customers, were highlighted as key areas for improvement.Moving forward, Telstra is likely to implement corrective measures aimed at strengthening its network resilience, optimizing operational procedures, and enhancing communication channels to ensure the prompt and effective handling of emergency situations.

These actions are essential for safeguarding the reliability and integrity of telecommunications services, particularly during critical situations where lives may be at risk.In the post-incident report on the outage, Telstra's CEO Vicki Brady disclosed a critical oversight that contributed to the failure of the backup processes. It was revealed that Telstra had stored incorrect alternative numbers for eight emergency services in a secondary database, which are typically used for manual call transfers during emergencies.

This revelation sheds light on a fundamental flaw in Telstra's backup system, highlighting the importance of accurate data management and validation procedures, particularly for critical services such as emergency call handling. The incorrect storage of alternative numbers compromised the effectiveness of manual call transfers during the outage, exacerbating the impact of the incident.

Addressing this issue will likely involve implementing robust data verification mechanisms, regular audits of backup systems, and ensuring that accurate and up-to-date information is maintained for emergency services. By rectifying these deficiencies, Telstra can enhance the reliability and resilience of its backup processes, thereby minimizing the risk of similar failures in the future and ensuring the prompt and effective response to emergency situations.The incorrect numbers stored by Telstra "prevented our team from making the manual transfer of the call to the respective emergency services operator," as CEO Vicki Brady explained.

The technical trigger for the outage was a combination of factors, starting with an unexpected database outage that exposed an existing but previously unknown software fault. This fault was discovered at 3:30 am when there was a surge in registration requests from medical alert devices. While this traffic alone wasn't sufficient to cause issues, it coincided with other system activities, leading to connections to the database reaching their maximum limit.When the database outage occurred, it "triggered an existing but previously undetected software fault" that disrupted the calling line identification (CLI) system, hindering its recovery.

Additionally, a breakdown in communication transpired when Telstra opted to reach out to emergency services via email for the failed call transfers, further exacerbating the situation.Brady expressed regret over the errors with the contact number and email address, acknowledging that having accurate contact information for emergency services is fundamental and should have been ensured.

She emphasized that relying on email as a fallback in such a critical situation was far from ideal and led to an unacceptable delay. The team resorted to email as a last resort when the manual transfer backup failed.Following the incident, Telstra has made significant efforts to address the root causes and prevent similar occurrences in the future.

Brady reported that they have identified and reproduced the issue with the CLI failure, and they are currently testing a fix for the software fault, with deployment expected by April.Additionally, Telstra has collaborated with organizations managing medical alert devices to implement changes so that device registration is only initiated when necessary for an emergency call.

Moreover, the carrier has conducted a comprehensive review of its end-to-end approach for Triple Zero, along with its monitoring and alarming systems, to enhance its ability to promptly identify and respond to any issues that may arise in the future.