TechnologyOne incident
Intermittent Service Disruption for a subset of customers ANZ
TechnologyOne experienced a minor incident on August 31, 2026 affecting Ci Anywhere, lasting 4h 31m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 31, 2026, 01:12 AM UTC
We are investigating an issue impacting service for ANZ. A subset of customers in ANZ are experiencing intermittent issues. Due to the investigation, the next update will be provided in 60 minutes, or sooner if new information becomes available.
- monitoring Aug 31, 2026, 01:21 AM UTC
Our team has verified the implementation of a fix is complete for ANZ Region. We will monitor the logs for the next 2 hours to ensure no further customers are impacted.
- monitoring Aug 31, 2026, 01:32 AM UTC
We are continuing to monitor for any further issues.
- monitoring Aug 31, 2026, 03:29 AM UTC
We are continuing to monitor for another 2 hours to ensure no further customers are impacted.
- resolved Aug 31, 2026, 05:43 AM UTC
After considerable monitoring, this incident is now resolved. We will undertake a Post Incident Review and findings will be posted here on completion. We apologise for how you and your business may have been affected by this incident. Please contact the TechnologyOne Support team if you continue to experience issues.
- postmortem Sep 15, 2026, 01:14 AM UTC
### Incident Summary Between **31 August 2026 and 2 September 2026**, some customers experienced intermittent service disruptions, including slow application responsiveness, connection timeouts, and periods of Ci unavailability. The issue was caused by internal DNS resolution failures within an underlying CiA cluster, which affected communication between application services and backend caching components. Service stability has been restored and all affected environments are operating normally. ### Root Cause A significant increase in internal DNS requests related to Redis caching placed unexpected demand on the cluster's DNS service \(CoreDNS\). This resulted in memory exhaustion within CoreDNS, preventing it from scaling effectively to meet demand. As a result, some application services experienced delayed or failed DNS lookups, leading to connection timeouts and degraded application performance. ### Corrective Actions Taken To restore service and stabilise the platform, we: * Increased CoreDNS capacity by scaling the service. * Rebalanced workloads by moving affected environments to alternative CiA clusters. * Restarted impacted DNS services and closely monitored DNS performance until normal operation was confirmed. ### Preventative Actions * Optimising Redis connection management and DNS resolution behaviour to reduce DNS query volumes. * Reviewing and refining CoreDNS autoscaling policies, resource limits, and capacity thresholds. * Assessing long-term cluster capacity and workload distribution strategies. * Enhancing monitoring and alerting to identify and respond to DNS-related issues before customer impact occurs.