Service Disruption for a subset of customers ANZ

Update timeline

1. investigating Sep 01, 2026, 02:27 AM UTC

   We are investigating an issue impacting service for a subset of ANZ customers. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

2. investigating Sep 01, 2026, 02:43 AM UTC

   A subset of customers across multiple releases in the ANZ region are experiencing Null Token errors and degraded access to CI and CI Anywhere. Our team is currently investigating and troubleshooting the issue. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

3. investigating Sep 01, 2026, 03:35 AM UTC

   We are continuing to investigate the issue affecting a subset of customers across multiple releases in the ANZ region. Impacted users may continue to intermittently experience Null Token errors when accessing CI. Access to CiA has improved. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

4. investigating Sep 01, 2026, 04:31 AM UTC

   We are continuing to investigate the issue related to Null Token errors that a subset of users may be intermittently experiencing when accessing CI across multiple releases in the ANZ region. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

5. investigating Sep 01, 2026, 05:33 AM UTC

   We are still continuing to investigate the issue related to Null Token errors that a subset of users may be intermittently experiencing when accessing CI across multiple releases in the ANZ region. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

6. investigating Sep 01, 2026, 06:36 AM UTC

   We are continuing to investigate the issue related to Null Token errors that a subset of users may be intermittently experiencing when accessing CI and Time tabling & Scheduling across multiple releases in the ANZ and UK region. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

7. investigating Sep 01, 2026, 07:33 AM UTC

   We are working with the upstream provider to investigate the issue related to Null Token errors that a subset of users may be intermittently experiencing when accessing CI and Time tabling & Scheduling across multiple releases in the ANZ and UK region. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.

8. monitoring Sep 01, 2026, 08:25 AM UTC

   We have seen error rates reduce significantly and are continuing to monitor the platform closely. Current indicators show improved service stability, and our teams remain engaged to ensure performance continues to trend in the right direction. We will maintain enhanced monitoring through tomorrow and provide our next update by 9:00 AM AEST, or earlier if required.

9. monitoring Sep 01, 2026, 10:33 PM UTC

   We continue to observe very low error rates and stable performance. Monitoring remains in place, and our teams are continuing to closely observe the environment to ensure service stability is maintained. We will provide our next update by 2:00 PM AEST, or earlier if required.

10. resolved Sep 02, 2026, 04:02 AM UTC

    Following an extended period of stable performance and continued monitoring, this incident has now been resolved. Error rates have remained at expected levels, and no further customer impact has been observed. Our teams will continue to review the incident and implement any identified improvements as part of our standard post-incident process. We appreciate your patience and understanding while we worked to restore and validate service stability.

11. postmortem Sep 15, 2026, 01:15 AM UTC

    ### Incident Summary Between **31 August 2026 and 2 September 2026**, some customers experienced intermittent service disruptions, including slow application responsiveness, connection timeouts, and periods of Ci unavailability. The issue was caused by internal DNS resolution failures within an underlying CiA cluster, which affected communication between application services and backend caching components. Service stability has been restored and all affected environments are operating normally. ### Root Cause A significant increase in internal DNS requests related to Redis caching placed unexpected demand on the cluster's DNS service \(CoreDNS\). This resulted in memory exhaustion within CoreDNS, preventing it from scaling effectively to meet demand. As a result, some application services experienced delayed or failed DNS lookups, leading to connection timeouts and degraded application performance. ### Corrective Actions Taken To restore service and stabilise the platform, we: * Increased CoreDNS capacity by scaling the service. * Rebalanced workloads by moving affected environments to alternative CiA clusters. * Restarted impacted DNS services and closely monitored DNS performance until normal operation was confirmed. ### Preventative Actions * Optimising Redis connection management and DNS resolution behaviour to reduce DNS query volumes. * Reviewing and refining CoreDNS autoscaling policies, resource limits, and capacity thresholds. * Assessing long-term cluster capacity and workload distribution strategies. * Enhancing monitoring and alerting to identify and respond to DNS-related issues before customer impact occurs.