Resolved after 4 hours 11 minutes, affecting Orchestrator, on July 1, 2026.
## Customer impact Between **July 1, 2026 at 3:38 pm UTC and July 1, 2026 at 6:58 pm UTC**, a subset of customers experienced increased latency and degraded performance when using Orchestrator in **U.S. region**. Some requests may have timed out during this period. The customer-impacting period lasted about 40 minutes. ## Root cause During a routine service update, a frequently-used database query was improved for better performance. The update to the query was correct; however, it caused the database to generate a new execution plan for that query. This new plan was built using a non-representative dataset, so it was not well-suited to production-scale data. As a result, the query began executing inefficiently, which drove a sustained increase in database CPU. The elevated CPU led to the query timeouts, increased request latency, and job execution delays that customers experienced during the incident. ## Detection The issue was detected through an automated availability alert at July 1, 2026 at 3:23 pm UTC. A customer-impacting incident was declared on July 1, 2026 at 3:39 pm UTC. ## Response By July 1, 2026 at 3:39 pm UTC, the affected Orchestrator database was identified as needing additional capacity. At 3:41 pm UTC, the decision was made to scale the database and update the public status page to indicate degraded performance and possible timeouts. The first public status page update was posted at 3:50 pm UTC, noting increased latency and degraded performance for Orchestrator in the U.S. region. By 3:59 pm UTC, the database scaling had completed and latency was confirmed to have returned to normal. At 4:02 pm UTC, the incident was marked as mitigated. A monitoring update was posted to the public status page at 4:10 pm UTC, confirming that mitigation had been applied and performance was improving, while noting that users might still experience some degraded performance. Monitoring continued after mitigation. The incident was formally resolved at 8:01 pm UTC. ## Follow-up * Activating query changes independently of deployment, so execution plans are generated against representative production data. * Tooling to detect and clear inefficient execution plans automatically, so the database can regenerate a better one. * Monitoring to flag when a single query consumes a disproportionate share of database CPU, so we can act before performance is affected.
Affected components
Update timeline
- investigating Jul 01, 2026, 03:50 PM UTC
Increased latency and degraded performance in US, Investigation is in progress.
- investigating Jul 01, 2026, 03:57 PM UTC
Increased latency and degraded performance in US, Investigation is in progress.
- monitoring Jul 01, 2026, 04:02 PM UTC
Mitigation has been applied and performance is improving for the issue that impacted [impacted functionality / user symptom] in [Service] across [Region]. Impact: Users may still experience [remaining symptom], but this should continue to improve. We are monitoring closely to ensure stability.
- monitoring Jul 01, 2026, 04:10 PM UTC
Mitigation has been applied and performance is improving for the issue that impacted Orchestrator in US. Impact: Users may still experience degraded performance, but this should continue to improve. We are monitoring closely to ensure stability.
- monitoring Jul 01, 2026, 05:31 PM UTC
Performance has continued to improve for the issue impacting Orchestrator in the US, and latency has decreased. Impact: Some users may still experience intermittent degraded performance while we continue to investigate. We are monitoring closely to ensure stability.
- monitoring Jul 01, 2026, 06:58 PM UTC
The issue has been mitigated and Orchestrator performance has returned to expected levels after degraded performance impacted US. Impact: No ongoing user impact.
- resolved Jul 01, 2026, 08:01 PM UTC
The regional outage has been resolved and all services in US are fully operational. Impact: No ongoing user impact.
- postmortem Jul 24, 2026, 10:18 AM UTC
## Customer impact Between **July 1, 2026 at 3:38 pm UTC and July 1, 2026 at 6:58 pm UTC**, a subset of customers experienced increased latency and degraded performance when using Orchestrator in **U.S. region**. Some requests may have timed out during this period. The customer-impacting period lasted about 40 minutes. ## Root cause During a routine service update, a frequently-used database query was improved for better performance. The update to the query was correct; however, it caused the database to generate a new execution plan for that query. This new plan was built using a non-representative dataset, so it was not well-suited to production-scale data. As a result, the query began executing inefficiently, which drove a sustained increase in database CPU. The elevated CPU led to the query timeouts, increased request latency, and job execution delays that customers experienced during the incident. ## Detection The issue was detected through an automated availability alert at July 1, 2026 at 3:23 pm UTC. A customer-impacting incident was declared on July 1, 2026 at 3:39 pm UTC. ## Response By July 1, 2026 at 3:39 pm UTC, the affected Orchestrator database was identified as needing additional capacity. At 3:41 pm UTC, the decision was made to scale the database and update the public status page to indicate degraded performance and possible timeouts. The first public status page update was posted at 3:50 pm UTC, noting increased latency and degraded performance for Orchestrator in the U.S. region. By 3:59 pm UTC, the database scaling had completed and latency was confirmed to have returned to normal. At 4:02 pm UTC, the incident was marked as mitigated. A monitoring update was posted to the public status page at 4:10 pm UTC, confirming that mitigation had been applied and performance was improving, while noting that users might still experience some degraded performance. Monitoring continued after mitigation. The incident was formally resolved at 8:01 pm UTC. ## Follow-up * Activating query changes independently of deployment, so execution plans are generated against representative production data. * Tooling to detect and clear inefficient execution plans automatically, so the database can regenerate a better one. * Monitoring to flag when a single query consumes a disproportionate share of database CPU, so we can act before performance is affected.
Your product runs on more than UiPath
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services