Microsoft Azure Outage History

Started Oct 29, 2025, 03:41 PM UTC· resolved Oct 30, 2025, 12:05 AM UTC

1. postmortem Oct 30, 2025, 12:05 AM UTC

   Read Azure Front Door's blog series in making progress on key resilience repairs: https://aka.ms/AzureFrontDoor/Resiliency-Part1 Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/air/YKYN-BWZ What happened? Between 15:41 UTC on 29 October and 00:05 UTC on 30 October 2025, customers and Microsoft services leveraging Azure Front Door (AFD) and Azure Content Delivery Network (CDN) experienced connection timeout errors and Domain Name System (DNS) resolution issues. From 18:30 UTC on 29 October 2025, as the system recovered gradually, some customers started to see availability improve – albeit with increased latency – until the system fully stabilized by 00:05 UTC on 30 October 2025. Affected Azure services included, but were not limited to: Azure Active Directory B2C, Azure AI Video Indexer, Azure App Service, Azure Communication Services, Azure Databricks, Azure Healthcare APIs, Azure Maps, Azure Marketplace, Azure Media Services, Azure Portal, Azure Sphere Security Service, Azure SQL Database, and Azure Static Web Apps. Other Microsoft services were also impacted, including Microsoft 365 (see: MO1181369), the Microsoft Communication Registry website, Microsoft Copilot for Security, Microsoft Defender (External Attack Surface Management), Microsoft Dragon Copilot, Microsoft Dynamics 365 and Power Platform (see: MX1181378), Microsoft Entra ID (Mobility Management Policy Service, Identity & Access Management, and User Management), Microsoft Purview, Microsoft Sentinel (Threat Intelligence), Visual Studio App Center, and customers’ ability to open support cases (both in the Azure Portal and by phone). What went wrong and why? Azure Front Door (AFD) and Azure Content Delivery Network (CDN) route traffic using globally distributed edge sites supporting customers as well as Microsoft services including various management portals. The AFD control plane generates customer configuration metadata that the data plane consumes for all customer-initiated operations including purge and Web Application Firewall (WAF) on the AFD platform. Since customer applications hosted on AFD and CDN can be accessed by their end users from anywhere in the world, these changes are deployed globally across all its edge sites to provide a consistent user experience. A specific sequence of customer configuration changes, performed across two different control plane build versions, resulted in incompatible customer configuration metadata being generated. These customer configuration changes themselves were valid and non-malicious – however they produced metadata that, when deployed to edge site servers, exposed a latent bug in the data plane. This incompatibility triggered a crash during asynchronous processing within the data plane service. This defect escaped detection due to a gap in our pre-production validation, since not all features are validated across different control plane build versions. Azure Front Door employs multiple deployment stages, and a configuration protection system to ensure safe propagation of customer configurations. This system validates configurations at each deployment stage and advances only after receiving positive health signals from the data plane. Once deployments are rolled out successfully, the configuration propagation system also updates a ‘Last Known Good’ (LKG) snapshot (a periodic snapshot of healthy customer configurations) so that deployments can be automatically rolled back in case of any issues. The configuration protection system waits for approximately a minute between each stage, completing on an average within 5-10 minutes globally. During this incident, the incompatible customer configuration change was made at 15:35 UTC, and was applied to the data plane in a pre-production stage at 15:36 UTC. Our configuration propagation monitoring continued to receive healthy signals – although the problematic metadata was present, it had not caused any issues. Because the data plane crash surfaced asynchronously, after approximately five minutes, the configuration passed through the protection safeguards and propagated to later stages. This configuration (with the incompatible metadata) completed propagation to a majority of edge sites by 15:39 UTC. Since the incompatible customer configuration metadata was deployed successfully to the majority of fleet with positive health signal, the LKG was also updated with this configuration. The data plane impact began in phases starting with our preproduction environment at 15:41 UTC, and replicated across all edge sites globally by 15:45 UTC. As the data plane impact started, the configuration protection system detected this and stopped all new and inflight customer configuration changes from being propagated at 15:43 UTC. The incompatible customer configuration was processed by edge servers, causing crashes across our various edge sites. This also impacted AFD’s internal DNS service, hosted on the edge sites of Azure Front Door, resulting in intermittent DNS resolution errors for a subset of AFD customer requests. This sequence of events was the trigger for the global impact on the AFD platform. This AFD incident on 29 October was not directly related to the previous AFD incident, from 9 October. Both incidents were broadly related to configuration propagation risk (inherent to a global Content Delivery Network, in which route/WAF/origin changes must be quickly deployed worldwide) but while the failure mode was similar, the underlying defects were different. Azure Front Door’s configuration protection system is designed to validate configurations and proceed only after receiving positive health signals from the data plane. During the AFD incident on 9 October (Tracking ID: QNBQ-5W8) that protection system worked as intended, but was later bypassed by our engineering team during a manual cleanup operation. During this AFD incident on 29 October (Tracking ID: YKYN-BWZ) the incompatible customer configuration metadata progressed through the protection system, before the delayed asynchronous processing task resulted in the crash. Some of the learnings and repair items from the earlier incident are applicable to this incident as well, and are included in the list of repairs below. How did we respond? The issue started at 15:41 UTC and was detected by monitoring at 15:48 UTC, prompting our investigation. By 15:43 UTC the configuration protection system activated in response to widespread data plane issues, and automatically blocked all new and in-flight configuration changes from being deployed worldwide. Since the latest ‘last known good’ (LKG) version was updated with the conflicting metadata, we chose not to revert to it. To ensure system stability, we decided not to rollback to prior versions of the LKG either. Instead, we opted to edit the latest LKG, by removing the problematic customer configurations manually. We also opted to block all customer configuration changes from propagating to the data plane at 17:30 UTC so that, as we mitigate, we would not reintroduce this issue. At 17:40 UTC we began deploying the updated LKG configuration across the global fleet. Recovery required reloading all customer configurations at every edge site and rebalancing traffic gradually, to avoid overload conditions as Edge sites returned to service. This deliberate, phased recovery was necessary to stabilize the system while restoring scale and ensuring no recurrence of the issue. Many downstream services that use AFD were able to failover to prevent further customer impact, including Microsoft Entra and Intune portals, and Azure Active Directory B2C. In a more complex example, Azure Portal leveraged its standard recovery process to successfully transition away from AFD during the incident. Users of the Portal would have seen limited impact during this failover process and then been able to use the Portal without issue. Unfortunately, some services within the Portal did not have an established fallback strategy and therefore parts of the Portal experience continued to experience failures even after Portal recovery (for example, Marketplace). Timeline of major incident milestones: • 15:35 UTC on 29 October 2025 – Corrupt metadata first introduced, as described above. • 15:41 UTC on 29 October 2025 – Customer impact began, triggered by the resulting crashes. • 15:43 UTC on 29 October 2025 – Configuration protection system activated in response to issues. • 15:48 UTC on 29 October 2025 – Investigation commenced following monitoring alerts being triggered. • 16:15 UTC on 29 October 2025 – Focus of the investigation became examining AFD configuration changes. • 16:18 UTC on 29 October 2025 – Initial communication posted to our public status page. • 16:20 UTC on 29 October 2025 – Targeted communications to impacted customers sent to Azure Service Health. • 17:10 UTC on 29 October 2025 – Began updating the ‘last known good’ LKG configuration to remove problematic configurations manually. • 17:26 UTC on 29 October 2025 – Azure Portal failed away from Azure Front Door. • 17:30 UTC on 29 October 2025 – Blocked all customer configuration propagation to the data plane, in preparation for deploying the new configuration. • 17:40 UTC on 29 October 2025 – Initiated the deployment of our updated ‘last known good’ configuration. • 17:50 UTC on 29 October 2025 – Last known good configuration available to all Edge sites, which began gradually reloading LKG configuration. • 18:30 UTC on 29 October 2025 – AFD DNS servers recovered, allowing us to rebalance traffic manually to a small number of healthy Edge sites. Customers began seeing improvements in availability. • 20:20 UTC on 29 October 2025 – As a sufficient number of Edge sites had recovered, we switched to automatic traffic management – customers continued to see availability improve. • 00:05 UTC on 30 October 2025 – AFD impact confirmed mitigated for customers, as availability and latency had returned to pre-incident levels. Post mitigation, we temporarily blocked all AFD customer configuration changes at the Azure Resource Manager (ARM) level to ensure the safety of the data plane. We also implemented additional safeguards including (i) fixing the control plane and data plane defects, (ii) removing asynchronous processing from the data plane, (iii) introducing an additional ‘pre-canary’ stage to test customer configuration (iv) extending the bake time during each stage of the configuration propagation, and (v) improvements to the data plane recovery time from approximately 4.5 hours to approximately one hour. We began draining the customer configuration queue from 2 November 2025. Once these safeguards were fully implemented, this restriction was removed on 5 November 2025. The introduction of new stages in the configuration propagation pipeline was coupled with additional ‘bake time’ between stages – which has resulted in an increase in configuration propagation time, for all operations including create, update, delete, WAF operations on AFD platform, and cache purges. We continue to work on platform enhancements to ensure a robust configuration delivery pipeline and further reduce the propagation time. For more details on these temporary propagation delays, refer to http://aka.ms/AFD_FAQ. How are we making incidents like this less likely or less impactful? To prevent issues like this, and improve deployment safety... • We have fixed both the original control plane incompatibility, and the data plane bug described above. (Completed) • We are now enforcing complete synchronous processing of each customer configuration, before advancing to production stages. (Completed) • We have implemented additional stages in our phased configuration rollout, including extended bake time to help detect configuration related issues. (Completed) • We are decoupling configuration processing in data plane servers from active traffic-serving instances to isolated worker process instances, thereby removing the risk of any configuration defect impacting data plane processing. (Estimated completion: January 2026) • Once our pre-validation of this configuration pipeline is in place, we will work towards reducing the propagation time from 45 minutes to approximately 15 minutes. (Estimated completion: January 2026) • We are enhancing our testing and validation framework to ensure backwards compatibility with configurations generated across previous versions of the control plane build. (Estimated completion: February 2026) To reduce the blast radius of potential future issues... • We have migrated critical first-party infrastructure (including Azure Portal, Azure Communication Services, Marketplace, Linux Software Repository for Microsoft Products, Support ticket creation) into an active-active solution with fail away. (Completed) • In the longer term, we are enhancing our customer configuration and traffic isolation, to ensure that no impact to any other customers from single customers’ traffic or configuration issue – utilizing ‘micro cell’ segmentation of the AFD data plane. (Estimated completion: June 2026) To be able to recover more quickly from issues... • We have made changes to accelerate our data plane recovery time, by leveraging the local customer configuration caching more effectively – to restore customer configurations within one hour. (Completed) • We are making investments to reduce data plane recovery time further, to restore customer configurations within approximately 10 minutes. (Estimated completion: March 2026) To improve our communications and support... • We have addressed delays in delivering alerts via Azure Service Health to impacted customers, by making immediate improvements to increase resource thresholds. (Completed) • We will expand the automated customer alerts sent via Azure Service Health, to include similar classes of service degradation – to notify impacted customers more quickly. (Estimated completion: November 2025) • Finally, we will resolve the technical and staffing challenges that prevented Premier and Unified customers from being able to create a support request, by ensuring that we can failover to our backups systems more quickly. (Estimated completion: November 2025) How can customers make incidents like this less impactful? • Understand best practices for building a resilient global HTTP ingress layer, to maintain availability during regional or network disruptions: https://learn.microsoft.com/azure/architecture/guide/networking/global-web-applications/mission-critical-global-http-ingress • As an alternative for when management portals are temporarily inaccessible, customers can consider using programmatic methods to manage resources – including the REST API (https://learn.microsoft.com/rest/api/azure) and PowerShell (https://learn.microsoft.com/powershell/azure) • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/YKYN-BWZ

Read the full incident report →

Started Oct 09, 2025, 07:43 PM UTC· resolved Oct 09, 2025, 11:59 PM UTC

1. postmortem Oct 09, 2025, 11:59 PM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/air/QKNQ-PB8 What happened? Between 19:43 UTC and 23:59 UTC on 09 October 2025, customers may have experienced availability issues and failures when loading content for the Azure Portal as well as other management portals across Microsoft. Over the course of the incident, approximately 45% of customers using the management portals experienced some form of impact – with failure rates reaching their peak at approximately 20:54 UTC, after which time customers generally experienced availability improving when attempting to load content. Note that service management via programmatic methods (such as PowerShell or the REST API) and the availability of resources were not impacted. What went wrong and why? Various Microsoft management portals utilize backend services to host content for different portal extensions. Azure Front Door (AFD) and Content Delivery Network (CDN) services are used to accelerate delivery of application content. An earlier incident occurred (Tracking ID: QNBQ-5W8) impacting the availability of management portals primarily from Africa and Europe, with lesser impact to Asia Pacific and the Middle East. We invoked our Business Continuity Disaster Recovery (BCDR) protocol and updated our network filters to allow traffic to bypass AFD so that customer traffic could reach backend services hosting content. Automation scripts were used to update the traffic load balancing configuration to split traffic across multiple routes. However, the scripts inadvertently removed a configuration value, because the script used an API version that predated the introduction of that value. The AFD endpoint began failing its health checks because the monitor of the AFD route falsely interpreted it as unhealthy. As a result, it was no longer routing traffic. However, since traffic was routed through alternative paths, there was no impact to customers, and we were not aware that the configuration value was removed. These automation scripts had previously been used successfully in other scenarios. After recovery of the previous incident, we began taking steps to resume normal traffic through AFD by using these automation scripts. However, due to the previously mentioned configuration value, traffic was not served correctly. We updated network filters to allow customer traffic only through AFD and, as this propagated, the backend hosting services were no longer reachable via AFD. This resulted in the impact to customers starting at 19:43 UTC. How did we respond? • 11:59 UTC on 09 October 2025 – We performed operations to allow management portal traffic to bypass AFD. • 19:39 UTC on 09 October 2025 – After verifying recovery of the previous incident, we took steps to migrate management portal traffic back completely through AFD. • 19:43 UTC on 09 October 2025 – Management portal availability issues occurred and impact grew. • 19:44 UTC on 09 October 2025 – Initial automated alerts were triggered. • 19:50 UTC on 09 October 2025 – Engineers were engaged and began investigating. • 20:30 UTC on 09 October 2025 – We confirmed the issues were related to the traffic migration and continued to investigate contributing factors. • 20:54 UTC on 09 October 2025 – We recovered one hosting service domain to correctly resume traffic through AFD and purged its caches to speed up recovery. Gradual recovery happened over the next 60 minutes, though other residual impact continued. • 21:30 UTC on 09 October 2025 – Azure Portal availability was mitigated, however other residual impact remained for other management portals. We shifted our focus to investigating these other residual failures and mitigating this impact. • 23:59 UTC on 09 October 2025 – We recovered an additional hosting service domain to correctly resume traffic through AFD. After reviewing our configuration and monitoring telemetry, we confirmed mitigation. How are we making incidents like this less likely or less impactful? • We have audited our architecture for the traffic routing of our management portals. (Completed) • We have updated our AFD migration scripts to use the latest API versions, to ensure that future runs will not remove the problematic configuration value. (Completed) • We are updating our standard operating procedures to add additional validation steps, to help ensure confirmation that traffic is being routed as expected. (Estimated completion: October 2025) • We are hardening our client-side logic so that browsers can fallback from AFD and hosting service incidents. (Estimated completion: November 2025) • We are making improvements to our failover systems from AFD, to be more robust and automated, so no manual intervention is required. (Estimated completion: December 2025) • We will be conducting additional simulation drills on our internal environments, to ensure that traffic shifts exactly as expected. (Estimated completion: January 2026) • In the longer term, we will be updating the architecture of our infrastructure to support regional rollout of such network traffic changes, to reduce the potential impact radius. (Estimated completion: March 2026) How can customers make incidents like this less impactful? • As an alternative for when management portals are inaccessible, customers can consider using programmatic methods to manage resources – including the REST API (https://learn.microsoft.com/rest/api/azure) and PowerShell (https://learn.microsoft.com/powershell/azure) • During service incidents in which Azure Portal content is unavailable, customers can try using our preview Azure Portal endpoint URL: https://preview.portal.azure.com • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/QKNQ-PB8

Read the full incident report →

Started Oct 09, 2025, 07:50 AM UTC· resolved Oct 09, 2025, 04:00 PM UTC

1. postmortem Oct 09, 2025, 04:00 PM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/AIR/QNBQ-5W8 What happened? Between 07:50 UTC and 16:00 UTC on 09 October 2025, Microsoft services and Azure customers leveraging Azure Front Door (AFD) and Azure Content Delivery Network (CDN) may have experienced increased latency and/or timeouts – primarily across Africa and Europe, as well as Asia Pacific and the Middle East. This impacted the availability of the Azure Portal as well as other management portals across Microsoft. Peak failure rates for AFD reached approximately 17% in Africa, 6% in Europe, and 2.7% in Asia Pacific and the Middle East. Availability was restored by 12:50 UTC, though some customers continued to experience elevated latency. Latency returned to baseline levels by 16:00 UTC, at which point the incident was mitigated. What went wrong and why? AFD routes traffic using globally distributed edge sites and supports Microsoft services including the management portals. The AFD control plane generates system metadata that the data plane consumes for customer-initiated ‘create’, ‘update’, or ‘delete’ operations on AFD or CDN profiles. One of the trigger conditions for this incident was a software defect in the latest version of the AFD control plane which had been rolled out six weeks prior to the incident, in line with our safe deployment practices. Newly created customer tenant profiles were being onboarded to the newer control plane version. Our service monitoring detected elevated data plane crashes due to a previously unknown bug – triggered by erroneous metadata, generated by a particular sequence of profile update operations. Our automated protection layer intercepted this in early update stages and prevented this metadata from propagating any further to the data plane, thereby averting any customer impact at that time. In addition, as the newer control plane was running in tandem with the previous version of the control plane, we disabled the new control plane from taking any requests. On 09 October 2025, we initiated a cleanup of the affected tenant configuration with the erroneous metadata. Since the automated protection system was blocking the impacted customer tenant profile updates in the initial stage, we temporarily bypassed it to allow the cleanup of the tenant configuration to proceed. By bypassing the protection system, the erroneous metadata was inadvertently able to propagate to later stages – and triggered the bug in the data plane that crashed the data plane service. This resulted in a disruption to a significant number of edge sites across Europe and Africa, approximately 26% of AFD data plane infrastructure resources in these regions were impacted. As part of AFD mechanisms to manage traffic, load was automatically distributed to nearby edge sites (including in Asia Pacific and the Middle East). Additionally, as regional business hours traffic started ramping up, it added to the overall traffic load. The increased volume of traffic on the remaining healthy edge sites resulted in high resource utilization, which exceeded operational thresholds. This triggered an additional layer of protection which started distributing traffic to a broader set of edge sites globally, to reduce further impact. Recovery required a combination of automated restarts, manual intervention where automated restarts were taking too long, and traffic failover operations for impacted management portals. Full mitigation was achieved once edge site infrastructure resources stabilized and latency returned to normal. Additionally, initial customer notifications were delayed primarily due to challenges determining impact, while attempting to target communications to those impacted. We have automated communications to notify customers of incidents quickly, unfortunately this capability was not yet supported in this incident scenario. How did we respond? • 07:30 UTC on 09 October 2025 – The cleanup operation was initiated. • 07:50 UTC on 09 October 2025 – Initial customer impact began, and increased over the next 90 minutes. • 08:13 UTC on 09 October 2025 – Our telemetry detected resource availability loss across multiple AFD edge sites. We began investigating as impact continued to grow. • 09:04 UTC on 09 October 2025 – We identified that the crashes were due to the previously identified data plane bug. • 09:08 UTC on 09 October 2025 – Automated restarts began for our AFD infrastructure resources, and manual intervention began for resources that did not recover automatically. • 09:15 UTC on 09 October 2025 – Customer impact had grown to be at its peak. • 10:01 UTC on 09 October 2025 – Communications were published to the Azure Status page. • 10:45 UTC on 09 October 2025 – Targeted customer communications were sent to Azure Service Health. • 11:59 UTC on 09 October 2025 – Management portals, like the Azure Portal, performed failover operations (including using scripts to update the load balancing configuration, to split traffic between multiple routes) helping restore its service availability. • 12:50 UTC on 09 October 2025 – Availability for AFD fully recovered, however a subset of customers may still have been experiencing elevated latency. • 16:00 UTC on 09 October 2025 – After continuous monitoring of latency improvement, we declared the incident as mitigated after confirming recovery. How are we making incidents like this less likely or less impactful? • We have hardened our standard operating procedures, to ensure that the configuration protection system is not bypassed for any operation. (Completed) • We have fixed the control plane defect which generated the erroneous tenant metadata that led to the data plane resource crashes. (Completed) • We have fixed the bug in the data plane. (Completed) • We will expand the automated customer alerts sent via Azure Service Health, to include similar classes of service degradation. (Estimated completion: November 2025) • We are making improvements to our Azure Portal failover systems from AFD, to be more robust and automated. (Estimated completion: December 2025) • We are building additional runtime configuration validation pipelines against a replica of real-time data plane, as a pre-validation step prior to applying them broadly. (Estimated completion: March 2026) • We are improving data plane resource instance recovery time, following any impact to the data plane. (Estimated completion: March 2026) How can customers make incidents like this less impactful? • Consider implementing failover strategies with Azure Traffic Manager, to fail over from Azure Front Door to your origins: https://learn.microsoft.com/azure/architecture/guide/networking/global-web-applications/overview • Consider reviewing our best practices for Azure Front Door architecture: https://learn.microsoft.com/azure/well-architected/service-guides/azure-front-door • Consider implementing retry patterns with exponential backoff, to improve workload resiliency: https://learn.microsoft.com/azure/architecture/patterns/retry • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: http://aka.ms/AzPIR/QNBQ-5W8

Read the full incident report →

Started Sep 26, 2025, 11:54 PM UTC· resolved Sep 27, 2025, 09:59 PM UTC

1. postmortem Sep 27, 2025, 09:59 PM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/AIR/BT6W-FX0 What happened? Between 23:54 UTC on 26 September 2025 and 21:59 UTC on 27 September 2025, a platform issue resulted in an impact to customers in Switzerland North who may have experienced service availability issues for resources hosted in the region. Many customers experienced mitigation by 04:00 UTC on 27 September 2025, while those impacted by additional residual impacts that remained after initial mitigation steps experienced full mitigation as late as 21:59 UTC on 27 September 2025. Impacted services included Azure API Management, Azure App Service, Azure Application Gateway, Azure Application Insights, Azure Cache for Redis, Azure Cosmos DB, Azure Data Explorer, Azure Database for PostgreSQL, Azure Databricks, Azure Firewall, Azure Kubernetes Service, Azure Storage, Azure Synapse Analytics, Azure Backup, Azure Batch, Azure Data Factory, Azure Log Analytics, Azure SQL Database, Azure SQL Managed Instance, Azure Site Recovery, Azure Stream Analytics, Azure Virtual Machine Scale Sets (VMSS), Azure Virtual Machines (VMs), and Microsoft Sentinel. Other services reliant on these may have also been impacted. What went wrong and why? A planned configuration change was made to the certificates used to authorize the communication for our software load balancer infrastructure that routes traffic for services in the Switzerland North region. One of the new certificates contained a malformed value that was not identified during the validation process. Moreover, this specific configuration change went through an expedited path - which unexpectedly applied it across multiple availability zones in the region - without evaluation of system health safeguards. This type of planned change is intended to be impactless but, due to the malformed certificate, authorization of connectivity for load balancer between nodes and resources could not complete. This caused significant impact to storage resources in the region. When a VM loses connectivity to its underlying virtual disk for an extended period, the platform detects the issue and safely shuts down the VM until connection is re-established, to prevent data corruption. Impact magnified because of services dependent on these resources. We corrected the certificate error and deployed the fix to re-establish connectivity for load balancer within the region. Most affected resources automatically recovered once connectivity was restored. However, a subset of resources (such as SQL Managed Instance VMs, VMs using Trusted Launch configurations, some Service Fabric Managed Cluster based services, and services that depend on them) did not resume operation automatically. These remained in a stopped or unhealthy state. • For SQL Managed Instance VMs, after load balancer connectivity was restored, the VMs remained unhealthy. This was due to a race condition during VM startup where different service specific operations customized for this type of VM were not completed in the expected sequence. This required each impacted VM to have its services restarted. • For Trusted Launch VMs, which use Trusted Launch with guest state stored in remote blobs, a unique failure mode was experienced. This incident resulted in a shutdown signal that is also used when customers intentionally reboot their VMs. The platform misinterpreted these events as customer-initiated and did not trigger automated recovery. As a result, these VMs remained in a stopped state and required explicit manual intervention to restart. • For Service Fabric Managed Cluster based services, a specific custom extension is present which required a specific sync pattern to succeed in reaching the storage account in order to provision the VM. For a subset of resources, this timed out due to transient failures. Once the VMs were properly restarted, these recovered. How did we respond? • 23:54 UTC on 26 September 2025 – Customer impact began, triggered by the configuration change. • 00:08 UTC on 27 September 2025 – The issue was detected via our automated monitoring. • 00:12 UTC on 27 September 2025 – Investigation commenced by our Azure Storage and Networking engineering teams. • 02:33 UTC on 27 September 2025 – We performed steps to revert the configuration change for the Switzerland North region. • 03:40 UTC on 27 September 2025 – The certificate was successfully reverted for the region. • 04:00 UTC on 27 September 2025 – Validation of recovery was confirmed for the majority of impacted services, but a subset of resources were still unhealthy due to residual impact. We performed an extended health evaluation to confirm which remaining resources required recovery, and investigated the steps were needed to recover residual impact. • 06:19 UTC on 27 September 2025 – We took additional time to recover remaining residual impact which included developing and validating new scripts to start resources that did not recover automatically. • 11:24 UTC on 27 September 2025 – Performed necessary checks to test safe recovery actions for different types of resources. • 14:15 UTC on 27 September 2025 – Recovery operations were performed for residual impact. This included executing mitigation scripts, and performing steps to safely reboot and recover remaining resources. • 16:15 UTC on 27 September 2025 – Partial recovery for residual impact. Additional operations to recover the remaining impact included properly restarting the resources in a stopped state. • 21:59 UTC on 27 September 2025 – Residual impact recovery activities and validation completed, confirming full recovery of all impacted services and customers. How are we making incidents like this less likely or less impactful? • We have implemented additional auditing into our deployment systems, designed to de-risk issues like this one. (Completed) • We have extensively evaluated our deployment pipelines and removed expedited pipelines that could result in this class of issue. (Completed) • We have improved our deployment safety measures for such changes, with additional automated rollback capabilities at earlier stages. (Completed) • We are improving our monitoring signals, to better understand the health posture of different resources and to expedite recovery. (Estimated completion: November 2025) • In the longer term, we are investing in improvements to our recovery processes for residual impact after these events. In particular, we will apply robust improvements for different custom extension processes and improve the resiliency of different unique resource startup operations to avoid requiring manual intervention. How can customers make incidents like this less impactful? • For mission-critical workloads, customers should consider a multi-region geodiversity strategy to avoid impact from incidents like this one that impacted a single region: https://learn.microsoft.com/azure/well-architected/design-guides/regions-availability-zones#deployment-approach-4-multi-region-deployments and https://learn.microsoft.com/azure/well-architected/design-guides/disaster-recovery • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/BT6W-FX0

Read the full incident report →

Started Sep 10, 2025, 09:03 AM UTC· resolved Sep 10, 2025, 06:50 PM UTC

1. postmortem Sep 10, 2025, 06:50 PM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/AIR/VKY3-PF8 What happened? Between 09:03 UTC and 18:50 UTC on 10 September 2025, an issue with the Allocator service in the Azure Compute control plane, led to degraded service management operations for resources in two Availability Zones (physical AZ02 and AZ03) in the East US 2 region. While the Allocator issue was mitigated by 18:50 UTC, a subset of downstream services experienced delayed recovery due to backlogged operations and retry mechanisms, with full recovery observed by 19:30 UTC. During this impact window, customers may have received error notifications and/or been unable to start, stop, scale, or deploy resources. The primary services affected were Virtual Machines (VMs) and Virtual Machines Scale Sets (VMSS) but services dependent on them were also affected – including but not limited to: • Azure Backup: Between 13:32 UTC and 19:30 UTC on 10 September 2025, customers may have experienced failures with Virtual Machine backup operations. • Azure Batch: Between 09:50 UTC and 18:50 UTC on 10 September 2025, customers may have experienced pool operations getting stuck. Primarily, pool grow/shrink and deletion operations may have become stuck. The majority of impact occurred between 15:06 and 18:50 UTC, but backlogged operations would have caught up by 19:30 UTC. • Azure Databricks: Between 10:15 UTC and 18:07 UTC on 10 September 2025, customers may have experienced delays and/or failures with job run operations and/or Databricks SQL queries. • Azure Data Factory: Between 10:32 UTC and 18:08 UTC on 10 September 2025, customers may have experienced failures in running dataflow jobs due to cluster creation issues. • Azure Kubernetes Service: Between 09:05 UTC and 19:30 UTC on 10 September 2025, customers may have experienced failures for operations, including but not limited to - create, upgrade, scale up, or delete managed clusters or node pools, or create or delete certain pods which require scaling up nodes, or attach/detach disks to VMs. • Azure Synapse Analytics: Between 10:28 UTC and 18:21 UTC on 10 September 2025, customers may have experienced spark job execution failures while executing spark activities within notebooks, data pipelines, or SDKs. • Microsoft Dev Box: Between 12:30 UTC and 17:15 UTC on 10 September 2025, customers may have experienced operation failures, including manual start via Dev Portal and connecting directly via the Windows App to a Dev Box in a shutdown or hibernated state. While AZ01 was not affected by the allocator service issue that impacted AZ02 and AZ03, allocation failures in AZ01 were observed as result of existing constraints on AMD v4/v5 and Intel v5 SKUs. What went wrong and why? Azure Resource Manager (ARM) provides a management layer that enables customers to create, update, and delete resources such as VMs. As part of this service management system, ARM depends on several VM management services in the Azure control plane, which in turn rely on an Allocator service deployed per Availability Zone. A recent change to Allocator behavior—designed to improve how the service handles heavy traffic— had been progressively rolled out over the prior two months, expanding to broader regions in early September. This update introduced a new throttling pattern that had performed effectively elsewhere. However, a performance issue affecting a subset of Allocator machines triggered the new throttling logic which, because of the high throughput demand in the East US 2 region, caused VM provisioning requests to retry more aggressively than the control plane could handle. Customer deployments failing in AZ03 were automatically redirected to AZ02, which already had significant load. This redirection, combined with normal traffic, led to overload and throttling in AZ02. Unlike AZ03, which recovered quickly once new VM deployments were halted, AZ02 faced a more complex and persistent challenge. By the time we paused new VM deployments in AZ02, the zone had already accumulated a significant backlog—estimated to be many times larger than normal operational volume. This backlog wasn’t just a result of repeated retries of the same operation. Instead, the platform was attempting a wide variety of retry paths to find successful outcomes for customers. These included retries across multiple internal infrastructure segments within AZ02, triggered by disk and VM co-location logic, and retries with different allocation parameters. Each retry was an attempt to find a viable configuration, not a simple repetition of the same request. This behavior reflects a platform optimized for reliability—one that persistently attempts alternative solutions rather than failing fast. While this approach often helps customers succeed in constrained environments, in this case it led to a large, persistent queue of operations that overwhelmed control plane services in AZ02. The zone’s larger size and higher throughput compared to AZ03 further compounded the issue, making recovery significantly slower. Importantly, the majority of the incident duration was spent in the recovery phase for AZ02. Once new deployments were halted and retries suppressed, the platform still had to drain the backlog of queued operations. This was not a simple in-memory retry loop—it was a persistent queue of work that had accumulated across multiple services and layers. Recovery was further slowed by service-level throttling configurations that, while protective, limited how quickly the backlog could be processed. These throttling values were tuned for normal operations and not optimized for large-scale recovery scenarios. We acknowledge that the extended recovery time in AZ02 was driven by the scale and complexity of the accumulated work, the opportunistic retry behavior of the platform, and the limitations of current throttling and queue-draining mechanisms. Addressing these factors is a key focus of our repair actions—both to prevent excessive backlog accumulation in future incidents, and to accelerate recovery when it does occur. During this incident, our initial communications strategy relied heavily on automated notifications to impacted customers, triggered by monitoring signals for each impacted service. While this system informed many customers quickly, it currently supports only a subset of services and scenarios. As a result, some impacted customers and services were not notified automatically, relying on manual communications provided later during the incident. As the scope of impact widened, we recognized that our communications were not reaching all affected customers. To address this, we sent broad targeted messaging to all customers running VMs in East US 2 – and ultimately published to the (public) Azure Status page, to improve visibility and reduce confusion. Finally, we acknowledge that early updates lacked detail around our investigation and recovery efforts. We should have shared more context sooner, and we have corresponding communication-related repair actions outlined below to improve timeliness and clarity in future incidents. How did we respond? • 09:03 UTC on 10 September 2025 – Customer impact began in AZ03. • 09:13 UTC on 10 September 2025 – Monitoring systems observed a spike in allocation failure rates for Virtual Machines, triggering an alert and prompting teams to investigate. • 09:23 UTC on 10 September 2025 – Initial targeted communications sent to customers via Azure Service Health. Automated messaging gradually expanded as impact increased, and additional service alerts were triggered. • 09:30 UTC on 10 September 2025 – We began manual mitigation efforts in AZ03 by restarting critical service components to restore functionality, rerouting workloads from affected infrastructure, and initiated multiple recovery cycles for the impacted backend service. • 10:02 UTC on 10 September 2025 – Platform services in AZ02 began to throttle requests due to the accumulation of excessive load, following customer deployments being automatically redirected from AZ03, in addition to normal customer load in AZ02. • 11:20 UTC on 10 September 2025 – Platform-initiated retries caused excessive load in AZ02, leading to increased failure rates. • 12:00 UTC on 10 September 2025 – We stopped new VM deployments being allocated in AZ03. • 12:40 UTC on 10 September 2025 – The reduction in load in AZ03 enabled AZ03 to recover. Customer success rates for operations on VMs that were already deployed into AZ03 returned to normal levels. • 13:30 UTC on 10 September 2025 – After verifying that platform-initiated retries had sufficiently drained, we re-enabled new VM deployments to AZ03. We then stopped all new VM deployments in AZ02. • 13:30 UTC on 10 September 2025 – We continued to apply additional mitigations to AZ02 platform services with backlogs of retry requests. These included applying more aggressive throttling, draining existing backlogs, restarting services, and applying additional load management strategies. • 15:05 UTC on 10 September 2025 – Customer success rates for operations on VMs already deployed into AZ02 started increasing. • 16:36 UTC on 10 September 2025 – Broad targeted messaging sent to all customers with Virtual Machines in East US 2. • 16:50 UTC on 10 September 2025 – We started gradually re-enabling traffic in AZ02. • 17:17 UTC on 10 September 2025 – The first update was posted on the public Azure Status page. • 18:50 UTC on 10 September 2025 – After a period of monitoring to validate the health of services, we were confident that the control plane service was restored. • 19:30 UTC on 10 September 2025 – Remaining downstream services reported recovery, following backlog drainage and retry stabilization. Customer impact confirmed mitigated. How are we making incidents like this less likely or less impactful? • We have turned off the recently deployed Allocator behavior, reverting to the previous allocation throttling logic. (Completed) • We have already adjusted our service throttling configuration settings for platform services in the Availability Zones of East US 2, as well as other high-load zones/regions. (Completed) • We are tuning and creating additional throttling levers in control plane services, to help prevent similar increased intra-service call volume load from occurring. (Estimated completion: October 2025) • We are creating a set of internal tools to support the draining of increased backlog for faster service recovery from incident scenarios that require this. (Estimated completion: October 2025) • We are addressing identified gaps in our load testing infrastructure, to help detect these types of issues earlier – before rolling out production changes. (Estimated completion: October 2025) • We are working to improve our communication processes and tools to help eliminate delays notifying impacted customers and supplying actionable information in the future. (Estimated completion: December 2025) • In the longer term, we are improving our end-to-end load management practices, to take a more holistic view of the impacts of any throttling, prioritization, and bursts in demand. (Estimated completion: June 2026) How can customers make incidents like this less impactful? • Note that the 'logical' Availability Zones used by each customer subscription may correspond to different physical Availability Zones - customers can use the Locations API to understand this mapping, to confirm which resources run in the physical AZs referenced above: https://learn.microsoft.com/rest/api/resources/subscriptions/list-locations • For mission-critical workloads, customers should consider a multi-region geodiversity strategy to avoid impact from incidents like this one that impacted a single region: https://learn.microsoft.com/training/modules/design-a-geographically-distributed-application and https://learn.microsoft.com/azure/architecture/patterns/geodes • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/VKY3-PF8

Read the full incident report →

Started Mar 18, 2025, 11:20 PM UTC· resolved Mar 19, 2025, 12:30 AM UTC

1. postmortem Mar 19, 2025, 12:30 AM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/AIR/Z_SZ-NV8 What happened? Between 13:37 and 16:52 UTC on 18 March, and again between 23:20 UTC on 18 March and 00:30 UTC on 19 March, 2025, a combination of a third-party fiber cut, and an internal tooling failure resulted in an impact to a subset of Azure customers with services in our East US region. During the first impact window, immediately after the fiber cut, customers may have experienced intermittent connectivity loss for inter-zone traffic that included AZ03 - to/from other zones, or to/from the public internet. During this time, the traffic loss rate peaked at 0.02% for short periods of time. Traffic within AZ03, as well as traffic to/from/within AZ01 and AZ02, was not impacted. During the second impact window, triggered by the tooling issue, customers may have experienced intermittent connectivity loss – primarily when sending inter-zone traffic that included AZ03. During this time, the traffic loss rate peaked at 0.55% for short periods of time. Traffic entering or leaving the East US region was not impacted, but there was some minimal impact to inter-zone traffic from both of the other Availability Zones, AZ01 and AZ02. Note that the 'logical' availability zones used by each customer subscription may correspond to different physical availability zones. Customers can use the Locations API to understand this mapping, to confirm which resources run in this physical AZ, see: https://learn.microsoft.com/rest/api/resources/subscriptions/list-locations?HTTP#availabilityzonemappings. What went wrong and why? At 13:37 UTC on 18 March 2025, a drilling operation near one of our network paths accidentally struck fiber used by Microsoft, causing an unplanned disruption to datacenter connectivity within AZ03. When fiber cuts impact our networking capacity, our systems are designed to redistribute traffic automatically to other paths. In this instance, we had two concurrent failures happen – before the cut, a datacenter router in AZ03 was down for maintenance and was in the process of being repaired. This combination of multiple concurrent failures impacted a small portion of our diverse capacity within AZ03, leading to the potential for intermittent connectivity issues for some customers. At 13:55 UTC on 18 March 2025, we began mitigating the impact of the fiber cut by load balancing traffic and restoring some of the impacted capacity – customers would have started to see their services recover at this time. Additionally, after the fiber cut and failed isolation, at 14:16 UTC a linecard failed on another router further reducing overall capacity to AZ03. However, as traffic had been re-routed, this further reduction in capacity did not cause any additional customer impact. During the initial mitigation efforts outlined above, our auto-mitigation tool encountered a lock contention problem blocking commands on the impacted devices, failing to isolate all capacity connected to those devices. This failure left some of the impacted capacity un-isolated, and our system did not flag this failed isolation state. Due to some capacity being out of service from the fiber cut, this failed state was not immediately flagged in our systems as the down capacity was not carrying production traffic. At approximately 21:00 UTC, our fiber provider commenced recovery work on the damaged fiber. During the capacity recovery process, at 23:20 UTC, as a result of the failure to isolate all the impacted fiber capacity, as individual fibers were repaired, our recovery systems begin re-sending traffic to the devices connected to the un-isolated capacity, therefore, bringing them back into service without safe levels of capacity. This caused traffic congestion that impacted customers as described above. The traffic congestion within AZ03, due to the tooling failure, triggered an unplanned failure mode on a regional hub router that connects multiple datacenters. By design, our network devices attempt to contain congestive packet loss to capacity that is already impacted. Due to the encountered failure mode, this containment failed on a subset of routers – so congestion spread to neighboring capacity on the same regional hub router, beyond AZ03. This containment failure impacted a small subset of traffic from the regional hub router to AZ1 and AZ2. At this stage, all originally-impacted capacity from the third-party fiber cut was manually isolated from the network – mitigating all customer impact by 00:30 UTC on 19 March. At 01:52 UTC on 19 March the underlying fiber cut was fully recovered. At that time, we completed the test and restoration of all capacity to pre-incident levels by 06:50 UTC on 19 March. How did we respond? • 13:37 UTC on 18 March 2025 – Customer impact began, triggered by a fiber cut causing network congestion which led to customers experiencing packet drops or intermittent connectivity. Our monitoring systems identified the impact immediately, so our on-call engineers engaged to investigate. • 13:45 UTC on 18 March 2025 – Our fiber provider was notified of the fiber cut and prepared for dispatch. • 13:55 UTC on 18 March 2025 – Mitigation efforts began identifying the impacted datacenters and redirecting traffic to healthier routes. • 15:07 UTC on 18 March 2025 – All customers using the East US region were notified about connectivity issues, even if their services were not directly impacted. • 16:52 UTC on 18 March 2025 – Mitigation efforts were successfully completed. All devices affected by the fiber cut were isolated, all customer traffic was using healthy paths and not experiencing congestion. • 23:20 UTC on 18 March 2025 – Customer impact recommenced, due to a tooling failure during the capacity repair process of the initial fiber cut. • 00:30 UTC on 19 March 2025 – This impact was mitigated after isolating the capacity that was incorrectly added by the tooling failure as part of the recovery process. Customers and services would have experienced full mitigation. • 01:52 UTC on 19 March 2025 – The underlying fiber cut was fully restored. We continued to monitor our capacity during the recovery process. • 06:50 UTC on 19 March 2025 – Fiber restoration efforts were completed. The incident was confirmed as mitigated. How are we making incidents like this less likely or less impactful? • We are fixing the tooling failure that caused the devices to be restored to take traffic before they were production ready. (Estimated completion: May 2025) • We are expediting a capacity upgrade within the most impacted datacenter, ahead of a planned technology refresh for all datacenters within this region - to de-risk the impact of multiple concurrent failures. (Estimated completion: July 2025) • In the longer term, we are working to limit the scope of impact further – specifically, to prevent the failure of a device from spreading across availability zones. (Estimated completion: February 2026) How can customers make incidents like this less impactful? • For mission-critical workloads, customers should consider a multi-region geodiversity strategy to avoid impact from incidents like this one that predominantly impacted a single region: https://learn.microsoft.com/training/modules/design-a-geographically-distributed-application and https://learn.microsoft.com/azure/architecture/patterns/geodes • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues - by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/ash-alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/Z_SZ-NV8

Read the full incident report →

Started Mar 03, 2025, 04:22 PM UTC· resolved Mar 03, 2025, 06:37 PM UTC

1. postmortem Mar 03, 2025, 06:37 PM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/AIR/ZT0X-9B8 What happened? Between 16:22 and 18:37 UTC on 3 March 2025, a subset of customers experienced issues when connecting to Microsoft services through our Toronto edge point-of-presence (PoP) location. This impacted customers connecting to Microsoft 365 and/or Azure services - including but not limited to Virtual Machines, Storage, App Service, SQL Database, Azure Kubernetes Service and Azure Portal access. These resources remained available for customers when connections were completed through other network paths. What went wrong and why? Our global network connects the datacenters across Azure regions with our large network of edge locations. Our edge PoP location in Toronto has paired network devices for redundancy. At 10:58 UTC one of the devices experienced hardware failures – we have since determined this to be due to a memory issue on the line cards for the device – and our automation successfully removed the device from the rotation. While we were performing an initial investigation into the cause of device failure, at 16:22 UTC the redundant device also experienced a hardware failure, due to a similar memory issue on the line cards - the line cards restarted, and did not initialize properly after startup, to forward traffic. As the paired device had already been taken out of rotation, earlier that day, automation was not able to perform normal mitigating actions, so it required human intervention. As such, this redundant device remained in rotation and led to network packets being dropped. During our investigation and evaluation of mitigation options, we considered failing traffic out of this PoP altogether, however this would have been particularly impactful to some customer configurations – for example, those with single-homed ExpressRoute connections – so we opted to keep the location in service. Since the hardware failure was on part of the device, our network engineers isolated the faulty line card from the device, which restored connectivity at 18:37 UTC. After further investigation of the hardware failures, we determined the memory increases were triggered due to an increase in the number of network flows through the device, which reached a scaling limit for these line cards. These occurrences of memory exhaustion coincided with a recent unrelated datacenter network change for specific devices, which began on 27 February 2025. The change increased the routing scale, which has since been paused. We monitor for overall memory on the devices, however the memory on the line cards was not monitored specifically, which led to a delay in our overall response and mitigation processes. As such, upon our initial investigation the overall scope of impact was not evident, which delayed broad customer communications. How did we respond? • 16:22 UTC on 3 March 2025 – Customer impact began, as the redundant device began experiencing the hardware failure, while the paired primary device had already been removed from rotation. • 16:32 UTC on 3 March 2025 – Our automation attempted to bring the paired primary device back into rotation and, due to system load balancing after route convergence, the impact was marginally reduced. • 16:43 UTC on 3 March 2025 – First automated alert raised of potential issues, however the scope of impact was not evident from the alert or initial investigations. • 17:19 UTC on 3 March 2025 – Network engineers investigating the issue took steps to remove congestion from the network peer, which reduced impact. • 17:27 UTC on 3 March 2025 – Additional customer reports received, following delays in submitting support requests via the Azure Portal. • 17:46 UTC on 3 March 2025 – Additional engineering teams were engaged to investigate and mitigate. • 17:59 UTC on 3 March 2025 – We began steps to manually remove the redundant device from rotation. • 18:37 UTC on 3 March 2025 – We isolated the faulty line card from the device. Network traffic was shifted to healthy routes, connectivity was restored, and we monitored the service health to ensure customers had been mitigated. How are we making incidents like this less likely or less impactful? • We have scanned and analyzed for the memory issue across our network. (Completed) • We have updated the configuration on the device, to handle more connections. (Completed) • We have added monitoring for this hardware subcomponent memory utilization, to detect issues like this more quickly. (Completed) • We are improving our notification service for similar issues, to notify impacted customers as soon as our internal monitoring detects the issue. (Estimated completion: March 2025) • In the longer term, we are improving our network design to handle multiple device failures in our edge PoP locations. (Estimated completion for Toronto edge: December 2025) How can customers make incidents like this less impactful? • For ExpressRoute high availability, we recommend operating both primary and secondary connections of ExpressRoute circuits in active-active mode: https://learn.microsoft.com/azure/expressroute/designing-for-high-availability-with-expressroute • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/ZT0X-9B8

Read the full incident report →

Started Feb 25, 2025, 04:42 PM UTC· resolved Feb 26, 2025, 01:15 AM UTC

1. postmortem Feb 26, 2025, 01:15 AM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/AIR/TMS9-J_8 What happened? Between 16:42 UTC on 25 February 2025 and 01:15 UTC on 26 February 2025, a platform issue affected Microsoft Entra ID, which prevented customers from authenticating via Microsoft Entra ID using the following features: • Seamless Single Sign-On (Seamless SSO) https://learn.microsoft.com/entra/identity/hybrid/connect/how-to-connect-sso • Microsoft Entra Connect Sync https://learn.microsoft.com/entra/identity/hybrid/connect/whatis-azure-ad-connect The error was caused by DNS resolution failures when trying to access select Microsoft Entra endpoints related to hybrid identity scenarios. Users were not able to seamlessly sign in to apps via Entra ID and were asked to interactively authenticate. For a subset of clients, authentication was completely blocked. The incident was mitigated for 94% of impacted customers by 18:35 UTC on 25 February 2025, and full mitigation occurred for the remainder of impacted customers at 01:15 UTC on 26 February 2025. What went wrong and why? As part of an ongoing maintenance effort, a change was made which inadvertently removed an intermediate DNS record and associated traffic manager used in the mentioned scenarios. The maintenance effort aimed to clean up interim routing topology, introduced during Microsoft Entra adoption of IPv6 in prior years. The DNS record and Traffic Manager were intermediate infrastructure components in the resolution path for autologon.microsoftazuread-sso.com, which is a domain name used in the Microsoft Entra ID's Seamless SSO feature and in the Entra Connect Sync feature. No other authentication flows were affected. The traffic manager was removed as configuration incorrectly indicated its DNS record was not in use. Similar changes leverage a drift detection system to validate that production DNS zone provisioning reflects the desired configuration. Unfortunately, due to a defect in this safety check, the configuration file was not flagged as out-of-sync. As a result, the DNS record was not correctly identified as in-use, so its removal proceeded. The change was reviewed in an internal change tracking system and correctly identified as high risk. However, this second safety check failed because the potential impact was not accurately identified, leading to the change being mistakenly approved to proceed. The issue was partially mitigated at 18:35 UTC for 94% of affected tenants by restoring configuration of the hostname via a CNAME record. The original DNS record was an A record which does not require clients to recursively resolve a hostname. Certain clients (https://aka.ms/CNAMEs) using Kerberos authentication are sensitive to the hostname being an A record and could not form the Service Principal Name (SPN) correctly, causing 403 Forbidden errors or server timeouts. At 01:05 UTC on 26 February, the configuration was reverted to its exact previous state and the impact was fully mitigated at 01:15 UTC on 26 February. Additionally, initial response to the incident and customer notifications were delayed by misconfiguration of auto-engagement systems and monitors. A feature of our incident management tooling used to request assistance from teams was partially affected. The team was required to leverage backup mechanisms to establish a shared incident bridge and merge investigation workstreams. How did we respond? • 16:42 UTC on 25 February 2025 – Relevant DNS records were inadvertently removed. Gradual onset of impact as the 5-minute DNS Time to Live (TTL) expired. • 17:18 UTC on 25 February 2025 – Investigation started based on internal DNS reachability monitor failures. • 17:40 UTC on 25 February 2025 – We identified and isolated the change that introduced the failure. • 18:35 UTC on 25 February 2025 – Approximately 94% of the customer impact had been mitigated, as the DNS configuration related to this authentication scenario had been partially restored. This change allowed autologon.microsoftazuread.sso.com to resolve again. • 19:16 UTC on 25 February 2025 – First notification posted to Azure Status page banner. • 22:35 UTC on 25 February 2025 – Through customer reports, we identified a subset of affected tenants were still experiencing issues, manifesting as 403 Forbidden errors or time outs. • 01:05 UTC on 26 February 2025 – The configuration for the affected hostname was rolled back to last known good state using an A record. • 01:15 UTC on 26 February 2025 – Traffic fully reverted to regular patterns. How are we making incidents like this less likely or less impactful? • We addressed the misconfiguration affecting our incident response systems by updating our auto-engagement processes to ensure timely notifications and updates are provided throughout the incident lifecycle. (Completed) • We have reviewed all internal monitoring related to outside-in reachability for authentication scenarios, to ensure these trigger escalations at the appropriate severity. (Completed) • We are prioritizing the resolution of the defect in the configuration drift detection system, to de-risk scenarios like this one. (Completed) • We will conduct a technical investigation on the feasibility of introducing non-global (e.g. regionally scoped) endpoints for Seamless SSO authentication scenarios. (Estimated completion: April 2025) • An internal investigation is taking place to audit change management practices and re-verify any changes with a similar risk profile over the last 24 months. A change freeze for DNS and traffic management is in place until this effort is completed. (Estimated completion: March 2025) • Furthermore, we will be upgrading the drift detection system to include additional exhaustive checks of all endpoints used in authentication scenarios. (Estimated completion: April 2025) • Currently, Azure Service Health alerts can only be configured at the Azure subscription level. To enable us to better communicate with our customers, we plan to enable tenant-level Service Health notifications, so that certain communications sent to Tenant Administrators can trigger Azure Service Health alerts. (Estimated completion: October 2025) How can customers make incidents like this less impactful? • Use Managed Identities for Azure Resources in lieu of user accounts employed as service accounts, where possible. Where the use of Managed Identities is not possible, adopt an application-only access pattern, rather than relying on service accounts: https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview and https://learn.microsoft.com/entra/identity-platform/app-only-access-primer. • Reevaluate whether Seamless SSO is still required. For customers using Windows 10, Windows Server 2016, or later versions, single sign-on via Windows Primary Refresh Tokens provides a more reliable and secure alternative to Seamless SSO. • As a general best practice, leverage latest authentication libraries such as MSAL, which include safe-by-default implementations of end-to-end authentication flows that account for a range of failure scenarios. • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://docs.microsoft.com/azure/architecture/framework/resiliency • The impact times above represent the full incident duration, so they are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, ensure that the right people in your organization will be notified about any future service issues by configuring Azure Service Health alerts. These alerts can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/ash-alerts and specifically here to receive Entra ID email notifications: https://learn.microsoft.com/entra/identity/monitoring-health/howto-configure-health-alert-emails How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/TMS9-J_8

Read the full incident report →

Started Jan 08, 2025, 10:31 PM UTC· resolved Jan 11, 2025, 12:44 AM UTC

1. postmortem Jan 11, 2025, 12:44 AM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/AIR/PLP3-1W8 What happened? Between 22:31 UTC on 8 January and 00:44 UTC on 11 January 2025, a networking configuration change in a single availability zone (physical zone AZ01) in East US 2 resulted in customers experiencing connectivity issues, prolonged timeouts, connection drops, and resource allocation failures across multiple Azure services. Customers leveraging Private Link with Network Security Groups (NSG) to communicate with Azure services may have also been impacted. Services that were configured to be zonally redundant and leveraging VNet integration may have experienced impact across multiple zones. The 'logical' availability zones used by each customer subscription may correspond to different physical availability zones. Customers can use the Locations API to understand this mapping, to confirm which resources run in this physical AZ - https://learn.microsoft.com/en-us/rest/api/resources/subscriptions/list-locations?HTTP#availabilityzonemappings . Impacted services included: • Azure Databricks: between 22:55 UTC on 08 January and 15:00 UTC on 10 January, customers may have experienced issues launching clusters and serverless compute resources, leading to failures in dependent jobs or workflows. • Azure OpenAI: between 04:29 UTC on 09 January and 15:00 UTC on 10 January, customers may have experienced stuck Azure OpenAI training jobs, and failures in Finetune model deployments and other jobs. • Azure SQL Managed Instance, Azure Database for PostgreSQL flexible servers, Virtual Machines, Virtual Machine Scale Sets: between 22:31 UTC on 08 January and 22:19 UTC on 10 January, customers may have experienced service management operation failures. • Azure App Service, Azure Logic Apps, Azure Function apps: between 22:31 UTC on 8 January and 20:05 UTC on 10 January, customers may have received intermittent HTTP 500-level response codes, timeouts or high latency when accessing App Service (Web, Mobile, and API Apps), App Service (Linux), or Function deployments hosted in East US 2. • Azure Container Apps: between 22:31 UTC on 08 January and 21:01 UTC on 10 January, customers may have experienced intermittent failures when creating revisions, container app/jobs in a consumption workload profile, and scaling out a Dedicated workload profile in East US 2. • Azure SQL Database: between 01:29 UTC on 09 January and 22:25 UTC on 10 January, customers may have experienced issues accessing services, and errors/timeouts for new database connections. Existing connections remained available to accept new requests, however re-established connections may have failed. • Azure Data Factory and Azure Synapse Analytics: between 22:31 UTC on January 8 and 00:44 UTC on January 11, customers may have encountered errors when attempting to create clusters, job failures, timeouts during Managed VNet activities, and failures in activity runs for the ExecuteDataFlow activity type. Other impacted services included API Management, Azure DevOps, Azure NetApp Files, Azure RedHat OpenShift, and Azure Stream Analytics. What went wrong and why? The Azure PubSub service is a key component of the networking control plane, acting as an intermediary between resource providers and networking agents on Azure hosts. Resource providers, such as the Network Resource Provider, publish customer configurations during Virtual Machine or networking create, update, or delete operations. Networking agents (subscribers) on the hosts retrieve these configurations to program the hosts networking stack. Additionally, the service functions as a cache, ensuring efficient retrieval of configurations during VM reboots or restarts. This capability is essential for deployments, resource allocation, and traffic management in Azure Virtual Network (VNet) environments. Around 22:31 UTC on 08 January, during an operation to enable a feature, a manual recycling of internal PubSub replicas on certain metadata partitions was required. This recycling operation should have been completed in ‘series’ to ensure that data quorum for the partition is maintained. In this case, recycling was executed across the replicas in ‘parallel’, causing the partition to lose quorum. This resulted in the PubSub service to lose the indexing data stored on these partitions. Losing the indexing data on three partitions resulted in ~60% of network configurations not being delivered to agents from the control plane. This impacted all service management operations for resources with index data on the affected partitions, including the network resource provisioning for virtual machines in East US 2, Az01. Networking agents would have failed to receive updates on VM routing information for new and existing VMs that underwent any automated recovery. This manifested in a loss of connectivity affecting VMs that customers use, in addition to the VMs that underpin our PaaS service offerings. Multi-tenant PaaS services that used VNet integration had a dependency on the partitions for the networking provisioning state. After the index data was lost, the dependency on this data that no longer existed caused impact to these PaaS services to expand beyond the impacted zone. How did we respond? The issue was detected by monitoring on the PubSub service at 22:44 UTC, with impacted services raising alerts shortly thereafter. The PubSub team identified the trigger as the three partitions with indexing data loss due to the recycling. At 23:10 UTC, we started evaluating different options for data recovery. Several workstreams were evaluated including: • Rehydrating the networking configurations to the partitions from the network resource provider. • Restoring from backup to recover the networking configurations. • Recovering data from the unaffected partitions. Due to limitations in feasibility for first two options, we started to recover data by rebuilding the mappings form the backend data store. Scanning the data store to extract source mappings is a time intensive process. It involves not only scanning the data, but also making sure that the mappings to the existing resources are still valid. To achieve this, we had to develop validation and mapping transformation scripts. We also throttled the mappings extraction to avoid impacting the backend data store. This scanning process took about 5-6 hours to complete. Once we obtained these mappings, we then had to restore the impacted index partitions. As we initiated the build phase, there were unforeseen challenges that we had to work around. Typically, each build takes 3-4 hours. We spent some time looking for ways to expedite this process, as our plan involved applying multiple builds. Independently on the builds taking longer than expected, we also observed a high number of failure requests due to the outage itself. As a result, our systems were generating a lot of errors which contributed to high CPU usage. This exacerbated the slowness in the restore process. One of the ways that we overcame this this was to deploy patches to block requests to the impacted partitions which mitigated the high CPU usage. Unfortunately, at 3:36 UTC on 10 January 2025, we discovered the first mapping source files that we generated were insufficient as we referenced an incorrect endpoint. Once this was understood, we had to repeat the mapping file generation and restore process. By 19:18 UTC on 10 January, all three partitions were fully recovered. Additional workstreams that were executed while this main workstream was underway included: • An immediate remediation step was to direct the Azure control plane service running on all host machines in this region to PubSub endpoints located in other non-impacted availability zones. This restored the host agent’s ability to look up the correct routing information resulting in mitigating VM to VM connectivity issues. • We initiated a workstream to route all new allocations requests to other availability zones in East US 2. However, as mentioned, the services that were using VNet integration, which had a fixed dependency on the impacted zone, would have continued to see failures. • To remediate impact to customers leveraging Private Link with NSGs to communicate with Azure services, we redirected the Private Link mapping to avoid the dependency on the impacted zone. By 19:18 UTC on 10 January, all three impacted partitions were fully recovered. Following this, we started working with impacted services to validate their health and remediate any data gaps that we discovered. At 00:30 UTC on the 11 January, we initiated a phased approach to gradually enable Az01 to accept new allocation requests. By 00:44 UTC on 11 January, all services confirmed mitigation. We reintroduced Az01 into rotation fully at 04:30 UTC, at which point we declared full mitigation. The impact window for Azure Services varied between service. Below is a timeline of events: • 22:31 UTC on 08 January 2025 – Customer impact begins following the recycling of the PubSub partitions. • 22:44 UTC on 08 January 2025 – The issue is identified as an incorrect networking configuration. • 23:10 UTC on 08 January 2025: We start to evaluate options for the index data recovery to the PubSub service. • 23:43 UTC on 08 January 2025: Targeted customer communications from the earliest reported impacted services started going out via Azure Service Health. • 00:35 UTC on 09 January 2025: We begin scanning the backend data store. • 02:37 UTC on 09 January 2025: Azure Status page updated with the details of a wider networking impact to the region. • 03:05 UTC on 09 January 2025: A workstream to route all new allocation requests to other availability zones in East US 2 is initiated. • 03:45 UTC on 09 January 2025: The workstream for the host agent’s ability to look up the correct routing information was completed. • 04:30 UTC on 09 January 2025: Developing scripts to restore the impacted partitions started. • 05:00 UTC on 09 January 2025: The workstream to route all new allocation requests to other availability zones in East US 2 was completed. • 07:45 UTC on 09 January 2025: Validation and testing of scripts for restoring mapping data to the impacted partitions is completed. • 08:51 UTC on 09 January 2025: Completed scanning backend generated source mapping files. • 09:14 UTC on 09 January 2025: Build process completed for the scripts to restore mappings. • 11:30 UTC on 09 January 2025: We observed high CPU and slow restoration speed. • 11:58 UTC on 09 January 2025: The workstream to mitigate impacted services with Private link and NSG dependencies is initiated. • 13:12 UTC on 09 January 2025: We updated one of the internal PubSub services to stop sending requests which was contributing to the high CPU usage. • 16:10 UTC on 09 January 2025: We completed another update to reduce request volumes to impacted partitions. This helped to improve the restore speed further. • 18:17 UTC on 09 January 2025: The workstream to redirect Private Link mapping to avoid the dependency on the impacted zone is completed. • 19:15 UTC on 09 January 2025: We created another patch to speed up the restore process. The new patch restores the mappings inside the partition rather than calling APIs to add mappings. • 23:40 UTC on 09 January 2025: This patching completed with the new optimization step, and the time for rebuilding one partition was reduced even further. Our overall restore time was reduced by >75%. • 02:10 UTC on 10 January 2025: Restoration of the first two partitions completed. • 03:00 UTC on 10 January 2025: Restoration of the third partition completed. • 03:36 UTC on 10 January 2025: Following the restoration, during our validation steps, we determined that the source mapping files generated were incorrect. As a result, we started generating all new source mapping files. • 05:00 UTC on 10 January 2025: We validated the data of source mappings for first partition. Continued extraction of the rest of the mappings. Meanwhile, we had to clean the previously restored partitions. • 08:30 UTC on 10 January 2025: The correct source mapping file is ready, and we started the partition restore process. • 11:04 UTC on 10 January 2025: The first partition is back online while troubleshooting and mitigating data anomalies. • 19:18 UTC on 10 January 2025 – All three of the impacted partitions are fully recovered and validated – At this stage most customers and services would have seen full mitigation. • 00:30 UTC on 11 January 2025 – We initiated a phased approach to gradually enable the impacted zone for new allocations. • 00:44 UTC on 11 January 2025 - All services confirmed mitigation. How are we making incidents like this less likely or less impactful? • We are working on reducing the recovery time for any data loss scenarios by improving the PubSub data recovery mechanism. This will bring our RTO to under 2 hours (Estimated completion: February 2025). • While PubSub production changes already go through our Safe Deployment Practices (SDP), we have added a step with additional layers of scrutiny. These will include, but are not limited to, verifying rollback plans, ensuring error budgets compliance, and performance validations (Completed). • All manual configuration changes will be moved to an automation pipeline which will require elevated privileges. (Estimated completion: February 2025). • Significant impact to the Pubsub service can occur if certain operations are performed incorrectly. We will block these operations from being executed, except by designated experts (Estimated completion: January 2025). • We are re-examining zonal services to ensure that all downstream dependencies are zone resilient. (Estimated completion: October 2025) • During the incident, we recommended that customers invoke a failover. This was an inappropriate communication. We are making corrections to our processes to ensure that our communications inform customers about the state of the impacted services, zones and regions. This, coupled with any other information that we have about the incident, including ETAs, should be taken into consideration as customers evaluate their own business continuity decisions (Estimated completion: January 2025) How can customers make incidents like this less impactful? • Consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://docs.microsoft.com/azure/architecture/framework/resiliency • Consider ensuring that the right people in your organization will be notified about any future service issues - by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/ash-alerts • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/PLP3-1W8

Read the full incident report →

Started Dec 26, 2024, 06:40 PM UTC· resolved Dec 27, 2024, 07:30 PM UTC

1. postmortem Dec 27, 2024, 07:30 PM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/AIR/CNF4-N_0 What happened? Between 18:40 UTC on 26 December and 19:30 UTC on 27 December 2024, multiple Azure services were impacted by a power event that occurred in one datacenter, within one Availability Zone (physical zone AZ03), in the South Central US region. Within the impacted datacenter, our automated power systems managed the event as expected, without interruption for two of the three data halls. However, one data hall did not successfully transition to an alternate power supply. This failure led to a loss of compute, network, and storage infrastructure in this data hall. Customer workloads configured for multi-zone resiliency would have seen no impact, or only brief impact, as automated mitigations occurred. Only customer workloads without multi-zone resiliency, and with dependencies on the impacted infrastructure, became degraded or unavailable. Impacted downstream services included: • Azure Alerts Management – between 18:40 UTC 26 December 2024 and 05:00 UTC on 27 December 2024, impacted customers may have experienced high latency in alerts notifications and persistence. • Azure App Service – between 18:40 UTC on 26 December and 12:00 UTC on 27 December, impacted customers may have received intermittent HTTP 500-level response codes, experience timeouts or high latency when accessing App Service (Web, Mobile, and API Apps), App Service (Linux), or Function deployments hosted in the South Central US region. • Azure Application Gateway – between 18:40 UTC on 26 December and 06:58 UTC on 27 December, impacted customers may have experienced data plane disruptions when trying to access your backend applications using Application Gateway hosted in the South Central US region. • Azure Backup – between 20:40 UTC on 26 December and 02:21 UTC on 27 December, impacted customers may have experienced failures in backup operations for Azure File shares in the South Central US region. • Azure Cache for Redis – between 18:45 and 21:35 UTC on 26 December, impacted customers may have lost cache availability and/or been unable to connect to cache resources hosted in the South Central US region. • Azure Cosmos DB – between 18:47 UTC on 26 December and 03:59 UTC on 28 December, impacted customers may have experienced a degradation in service availability and/or request latency. Some requests may have resulted in server errors or timeouts. • Azure Database for PostgreSQL – between 18:48 UTC on 26 December and 13:12 UTC on 27 December, impacted customers may have experienced connectivity failures and timeouts when executing operations, as well as unavailability of resources hosted in the South Central US region. • Azure Database Migration Service – between 19:01 UTC on 26 December and 12:17 UTC on 27 December, impacted customers may have experienced timeout errors when attempting to create a new migration service, or when using existing migration service in the South Central US region. • Azure Event Hubs | Azure Service Bus – Customers with Standard SKU, Premium SKU namespaces or AZ-enabled dedicated Event Hubs clusters experienced an availability drop for approximately five minutes, at the time when the incident started – this issue was mitigated automatically once namespace resources were reallocated to other availability zones. However, a subset of customers using Event Hubs Dedicated non-AZ clusters experienced an availability issue for an extended period of time when trying to access their Event Hubs namespaces in the region. The affected Event Hubs dedicated clusters recovered once the underlying failing VMs in their clusters were brought back online, the last of which were restored by 05:52 UTC on 27 December. • Azure Firewall – between 18:44 UTC on 26 December and 11:30 UTC on 27 December, impacted customers with an Azure Firewall deployed with multi-zone resilience may have seen partial throughput degradation and no availability loss. Customers with an Azure Firewall not utilizing multi-zone resiliency may have had resources dependent on the impacted Availability Zone (physical zone AZ03) which could have resulted in performance degradation or availability impact. Customers attempting control plane operations (for example, making changes to Firewall policies/rules) may have experienced failures during this incident. • Azure Logic Apps – between 18:47 UTC on 26 December and 03:10 UTC on 27 December, impacted customers may have encountered delays in run executions and failing data or control plane calls. • Azure SQL Database – between 20:12 UTC on 26 December and 18:22 UTC on 27 December, impacted customers may have experienced issues accessing services. New connections to databases in the South Central US region may have resulted in an error or timeout. Existing connections remained available to accept new requests, however if those connections were terminated then re-established, they may have failed. • Azure Storage – between 18:45 UTC on 26 December and 08:50 UTC on 27 December, impacted customers may have experienced timeouts and failures when accessing storage resources hosted in the South Central US region. This affected both Standard and Premium tiers of Blobs, Files and Managed Disks. • Azure Synapse Analytics – between 18:53 UTC on 26 December and 13:52 UTC on 27 December, impacted customers may have experienced spark job execution failures, and/or errors when attempting to create clusters, in the South Central US, East US 2, and/or Brazil South regions. • Azure Virtual Machines – between 18:41 UTC on 26 December and 22:26 UTC on 27 December, impacted customers may have experienced connection failures when trying to access some Virtual Machines hosted in the South Central US region. These Virtual Machines may have also restarted unexpectedly. • Azure Virtual Machine Scale Sets – between 19:04 UTC on 26 December and 11:18 UTC on 27 December, impacted customers may have experienced error notifications when performing service management operations - such as create, delete, update, scaling, start, stop - for resources hosted in the South Central US region. • This incident also impacted a subset of Microsoft 365 services – further details are provided in the Microsoft 365 Admin Center, under incident ID MO966473. What went wrong and why? This incident was initially triggered by a utility power loss, itself caused by a localized ground fault – in which a high voltage underground line failed. After a phase to ground short developed in the buried feeder cables, the breaker feeding the datacenter tripped – leading to a loss of utility power, at 18:40 UTC. By design, the power distribution systems transferred power to diesel backup generators, where UPS batteries carry the load during this transition, which was successful for two of the three affected data halls. During the transition to generator power, the third data hall experienced UPS battery faults, which caused the load to drop during transition. In any power-related event, our first priority is to ensure the safety of our staff and infrastructure before any power restoration work can begin. Following our assessment, we were able to safely begin restoration at 20:13 UTC. IT power loads were manually re-energized on backup diesel generator power, by performing a bypass on the failed UPS devices. We began seeing infrastructure services returning by 20:35 UTC, with power fully restored by 20:56 UTC. As power and infrastructure recovered, the next validation steps were to ensure that Azure Networking and Azure Storage services were recovering as expected. By 21:00 UTC, almost all storage and network infrastructure services were confirmed as fully operational. A single storage scale unit remained significantly degraded, due to hardware that required deeper inspection and ultimately, replacement. As storage scale units recovered, 85% of the impacted Virtual Machines (VMs) recovered by 21:40 UTC as their Virtual Hard Disks (VHDs) became available. The next 13% of VMs recovered between 06:00 and 06:30 UTC, as the final storage scale unit became available. Despite all the storage issues being resolved, <2% of VMs impacted by this event remained unhealthy. These issues are detailed below and explain why impacted downstream services with dependencies on these VMs experienced long-tail recoveries. The incident was declared as mitigated at 19:30 UTC on 27 December 2024. Azure Storage: For Zone Redundant Storage (ZRS) accounts, there was no availability impact – as data was served from replicas in other Availability Zones during this incident. The power loss event impacted six Storage scale units. After power restoration, scale units hosting Standard SSD Managed Disks, Premium SSD Managed Disks, Premium Blobs, and Premium Files, fully recovered automatically in around 30 minutes. For most of the HDD-based Standard Storage LRS/GRS scale units, the storage services took approximately one hour to recover. Unfortunately, within one Standard Storage scale unit, multiple network switches were non-functional following the power event, causing a significant portion of the data in that scale unit to be inaccessible because all replicas were unreachable. This caused significant impact to VMs and dependent services that were using Standard HDD managed disks and LRS blob or file storage accounts hosted on this scale unit. Mitigation required replacement networking equipment to be sourced from spares and installed by datacenter technicians. Network engineers then configured and validated these devices, before bringing them online. Additional actions were taken to recover storage nodes under the replaced switches. For the majority of accounts availability was restored by 06:10 UTC on 27th December 2024 (overall availability at 99.5%), with repairs required on a handful of servers to restore 100% availability by 08:50 UTC on 27 December 2024. Azure Compute / Virtual Machines: For customers using VM/compute workloads that leveraged multi-zone resiliency (such as VMSS flex across availability zones), there was no availability impact. For incidents like this, Azure has an automated recovery suite called ‘Defibrillator’ that starts automatically, to recover the VMs and Host machines they are running on, after datacenter power has been restored. It will orchestrate the power on for all affected Host machines, monitor the boot-up and bootstrap sequences, and ensure that the VMs are up and running. When this is running, Azure’s automated steady-state health detection and remediation systems suspend all activities, in order to avoid disrupting the disaster recovery process. At approximately 22:00 UTC on 26 December 2024, some compute scale units were found not tracking at the expected level of recovery. For the final 2% of VMs mentioned above, these experienced an extended recovery – we observed three separate events that contributed to this. • The first scenario was due to initialization without a connection to a network device. Due to the network devices not being fully configured before the Host machines were powered on, a race condition triggered during the Host bootstrap process. This issue is specific to a certain hardware configuration within localized compute scale units, and necessitated the temporary disabling of some validation checks during the bootstrap process. • The second scenario delaying recovery was some machines failing to boot into the Host OS due to a newly discovered bootloader bug impacting a small subset of host hardware with higher levels of offline memory pages. When the hardware reports repeated corrected memory errors to the Host OS, the Host will offline certain memory ranges to prevent repeated use of that memory range. In a small subset of host hardware where a large range of offline memory was accumulated, this new Host OS bug was discovered – resulting in failing to bootstrap the Host OS. This category was mitigated by clearing and/or ignoring this offline memory list and allowing the Host OS to make forward progress where it could, then rebuild its offline memory list once it started to run the full OS. • The third scenario that had prevented compute recovery in some cases was due to control plane devices that are inline to execute the power operations on the Host machines. Datacenter technicians were required to reseat that infrastructure manually. By 10:50 UTC on 27 December, >99.8% of the impacted VMs had recovered, with our team re-enabling Azure’s automated detection and remediation mechanisms. Some targeted remediation efforts were required for a remaining small percentage of VMs, requiring manual intervention to bring these back online. Azure Cosmos DB - For Azure Cosmos DB accounts configured with availability zones, there was no impact, and the account maintained availability for reads and writes. Impact on other Cosmos DB accounts varied depending on the customer database account regional configurations and consistency settings: • Database accounts configured with availability zones were not impacted by the incident, and maintained availability for reads and writes. • Database accounts with multiple read regions and a single write region outside South Central US maintained availability for reads and writes if configured with session or lower consistency. Accounts using strong or bounded staleness consistency may have experienced write throttling to preserve consistency guarantees until the South Central US region was either taken offline or recovered. This behavior is by design. • Active-passive database accounts with multiple read regions and a single write region in South Central US maintained read availability, but write availability was impacted until the South Central US region was taken offline or recovered. • Single-region database accounts in South Central US without Availability Zone configuration were impacted if any partition resided on the affected instances. Azure SQL Database: For Azure SQL Databases configured with zone redundancy, there was no impact. A subset of customers in this region experienced unavailability and slow/stuck control plane operations, such as updating the service level objective, for databases that are not configured as zone redundant. Customers with active geo-replication configuration were asked to consider failing out of the region at approximately 22:31 UTC. Impact duration varied. Most databases recovered after Azure Storage recovered. Some databases took an extended time to recover due to the aforementioned long recovery time of some underlying VMs. Azure Application Gateway: • Application Gateway experienced issues with data path, control plane, and auto-scale operations, leading to service disruptions. Impact on Application Gateways varied depending on customer configuration: • Customers who deployed Application Gateways with zone redundancy may have experienced latency issues and overall degraded performance. • Customers who deployed Application Gateways to a single zone or did not specify zone info during deployment may have experienced data path loss if their deployments had instances in the affected zone. • Gateways with instances deployed in affected zone may have experienced failures or delays in configuration updates. • Gateways with instances deployed in affected zone may have experienced failures or delays in auto scale operations. Azure Firewall: For Azure Firewalls deployed to all Availability Zones of the region, customers would not have experienced any data path impact. However, customers with an Azure Firewall deployed only to the impacted Availability Zone (physical zone AZ03), may have experienced some performance degradation – affecting the ability to scale out. Finally, customers attempting control plane operations (for example, making changes to Firewall policies/rules) may have experienced failures during this incident. Both of these impacts were experienced between 18:40 UTC on 26 December and 07:22 UTC on 27 December 2024. Azure Synapse: Some users of Azure Synapse Analytics faced spark job execution failures in South Central US, Brazil South, and EastUS2. This impacted less than 1% of Synapse calls in those regions. Your logs may include one or more of the following errors that could be a result of this issue: “CLUSTER_CREATION_TIMED_OUT”, “FAILED_CLUSTER_CREATION”, “CLUSTER_FAILED_AFTER_RUNNING”. During this period, Azure Synapse could not provision on-demand compute due to failure to retrieve Management Group's ancestry for RBAC evaluations. The underlying storage for the SCUS instance of the ancestry data was impacted by this incident, which South Central US, Brazil South, and EastUS2 regions depend on. The data is replicated globally and regional failover attempts were made, but did not succeed due to a gateway error. The issue was resolved across all regions once South Central US region was recovered. How did we respond? • 18:40 UTC on 26 December 2024 – Initial power event occurred which led to power loss in the affected data hall. • 18:45 UTC on 26 December 2024 – Technicians from datacenter operations team engaged. • 18:46 UTC on 26 December 2024 – Portal Communications started being sent to impacted subscriptions. • 19:02 UTC on 26 December 2024 – Datacenter incident call began to support triaging and troubleshooting issues. • 19:08 UTC on 26 December 2024 – Azure engineering teams joined a central incident call, to triage and troubleshoot Azure service impact. • 20:13 UTC on 26 December 2024 – Power restoration assessed safe and began. • 20:35 UTC on 26 December 2024 – Compute, Network, and Storage infrastructure began to recover. • 20:54 UTC on 26 December 2024 – Communications published to our public status page. • 20:56 UTC on 26 December 2024 – Power had been restored. Infrastructure recovery continued. • 21:40 UTC on 26 December 2024 – 85% of the VMs impacted by underlying VHD availability recovered. • 06:30 UTC on 27 December 2024 – Additional 13% of VMs impacted by VHD availability recovered. • 08:30 UTC on 27 December 2024 – Ongoing mitigation of additionally impacted services. • 13:00 UTC on 27 December 2024 – Mitigation to most affected services confirmed. • 19:30 UTC on 27 December 2024 – Incident mitigation confirmed and declared. How are we making incidents like this less likely or less impactful? • Datacenter response to return to utility power after ensuring battery health for UPS transition from generator (Completed). • We are reviewing the nature of the UPS battery failures in line with our global battery standards and maintenance procedures, to identify improvements to de-risk this class of issue across the fleet. (Estimated completion: February 2025) • Repairs to the offline failed utility line are in progress. (Estimated completion: February 2025) • The mitigation to bypass various checks during the bootstrap process have been applied to all impacted machines and are being evaluated and executed for other hardware configurations where needed. (Estimated completion: March 2025) How can customers make incidents like this less impactful? • Consider using Availability Zones (AZs) to run your services across physically separate locations within an Azure region. To help services be more resilient to datacenter-level failures like this one, each AZ provides independent power, networking, and cooling. Many Azure services support zonal, zone-redundant, and/or always-available configurations: https://docs.microsoft.com/azure/availability-zones/az-overview • Note that the 'logical' Availability Zones used by each customer subscription may correspond to different physical Availability Zones - customers can use the Locations API to understand this mapping, to confirm which resources run in this physical AZ - https://learn.microsoft.com/rest/api/resources/subscriptions/list-locations • Customers using Azure Virtual Machines should consider reviewing our reliability guidance, including Availability Zone support and disaster recovery. See: https://learn.microsoft.com/azure/reliability/reliability-virtual-machines#availability-zone-support • Customers using Azure Cosmos DB should consider reviewing our high availability and reliability guidance for adopting Availability Zone architecture: https://learn.microsoft.com/azure/reliability/reliability-cosmos-db-nosql • Customers using Azure SQL DB should consider reviewing our high availability and resiliency guidance surrounding relevant redundancy: https://learn.microsoft.com/azure/azure-sql/database/high-availability-sla-local-zone-redundancy?view=azuresql&tabs=azure-powershell • Customers currently using Application Gateway v1 should consider migrating to Application Gateway v2, see: https://learn.microsoft.com/azure/application-gateway/migrate-v1-v2 • Customers using Application Gateway v2 should consider ensuring that two or more Azure availability zones are selected during provisioning, for better resiliency. See: https://learn.microsoft.com/azure/application-gateway/overview-v2 • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://docs.microsoft.com/azure/architecture/framework/resiliency • Finally, consider ensuring that the right people in your organization will be notified about any future service issues - by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/ash-alerts How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/CNF4-N_0

Read the full incident report →

Started Nov 13, 2024, 12:50 AM UTC· resolved Nov 13, 2024, 12:30 PM UTC

1. postmortem Nov 13, 2024, 12:30 PM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/AIR/PSM0-BQ8 What happened? Between 00:50 UTC and 12:30 UTC on 13 November 2024, a subset of Azure Blob Storage and Azure Data Lake Storage accounts experienced connectivity errors. The issue manifested as loss of access to Blob and Data Lake storage endpoints of the affected storage accounts, and subsequent unavailability of downstream services that depended on these storage accounts. Since many of the impacted storage accounts were used by other Azure services and major software vendor solutions, the customer impact was widespread. Although unavailable to access, the data stored in these storage accounts was not impacted during this incident. Impacted downstream services included: • Azure Storage: Impacted customers may have experienced name (DNS) resolution failures when interacting with impacted storage accounts in Australia East, Australia Southeast, Brazil South, Brazil Southeast, Canada Central, Canada East, Central India, Central US, East Asia, East US, East US 2, East US 2 EUAP, France Central, Germany West Central, Japan East, Japan West, Korea Central, North Central US, North Europe, Norway East, South Africa North, South Central US, South India, Southeast Asia, Sweden Central, Switzerland North, UAE North, UK South, UK West, West Central US, West Europe, West US, West US 2, West US 3. • Azure Container Registry: Impacted customers using the East US region may have experienced intermittent 5xx errors while trying to pull images from the registry. • Azure Databricks: Impacted customers may have experienced failures with launching clusters and serverless compute resources in Australia East, Canada Central, Canada East, Central US, East US, East US 2, Japan East, South Central US, UAE North, West US, and/or West US 2. • Azure Log Analytics: Impacted customers using the West Europe, Southeast Asia, and/or Korea Central regions may have experienced delays and/or stale data when viewing Microsoft Graph activity logs. What went wrong and why? Azure Traffic Manager manages and routes blob and data lake storage API requests. The incident was caused by an unintentional deletion of Traffic Manager profiles for the impacted storage accounts. These Traffic Manager profiles were originally part of an Azure subscription pool which belonged to the Azure Storage service. This original service was bifurcated to become two separate services, with one service that would eventually be deprecated. Ownership of the subscriptions containing the Traffic Manager profiles for storage accounts should have been assigned to the service that was continuing operation, which was missed. As such, the decommissioning process inadvertently deleted the Traffic Manager profiles under the subscription, leading to loss of access to the affected storage accounts. To learn more about Azure Traffic Manager profiles, see: https://learn.microsoft.com/azure/traffic-manager/traffic-manager-manage-profiles. How did we respond? After receiving customer reports of issues, our team immediately engaged to investigate. Once we understood what had triggered the problem, our team initiated started to restore Traffic Manager profiles of the affected storage accounts. The recovery took an extended period of time since it required care reconstructing the Traffic Manager profiles while avoiding further customer impact. We started multiple workstreams in parallel, to drive both manual recovery and create automated steps to speed up recovery. Recovery was carried out in phases, with the majority of affected accounts restored by 06:24 UTC - and the last set of storage accounts recovered and fully operational by 12:30 UTC. Timeline of key events: • 13 November 2024 @ 00:50 UTC – First customer impact, triggered by the deletion of a Traffic Manager profile. • 13 November 2024 @ 01:24 UTC – First customer report of issues, on-call engineering team began to investigate. • 13 November 2024 @ 01:40 UTC – We identified that the issues were triggered by the deletion of Traffic Manager profiles. • 13 November 2024 @ 02:16 UTC – Impacted Traffic Manager profiles identified, and recovery planning started. • 13 November 2024 @ 02:25 UTC – Recovery workstreams started. • 13 November 2024 @ 03:51 UTC – First batch of storage accounts recovered and validated. • 13 November 2024 @ 06:00 UTC – Automation to perform regional recovery in place. • 13 November 2024 @ 06:24 UTC – Majority of recovery completed; most impacted accounts were accessible by this time. • 13 November 2024 @ 12:30 UTC – Recovery and validation 100% complete, incident mitigated. How are we making incidents like this less likely or less impactful? • We completed an audit of all the production service artifacts used by the Azure Storage resource provider. (Completed) • We created a new highly restrictive deployment approval process, as an additional measure to prevent unintended mutations like deletions. (Completed) • We are improving the process used to clean-up production service artifacts with built-in safety to prevent impact. (Estimated completion: December 2024) • We are enhancing our monitoring of outside-in storage traffic by making it more sensitive to smaller impacts and to validate connectivity and reachability for all endpoints of storage accounts. (Some services completed, all service will complete in December 2024) • We are expanding and completing the process of securing platform resources with resource locks, as an additional safety automation to prevent deletes. (Estimated completion: January 2025) • We will accelerate recovery times by refining restore points and optimizing the recovery process for production service artifacts. (Estimated completion: January 2025) • We will reduce the blast radius with service architectural improvements to improve the resiliency against issues related to traffic manager, and other upstream dependencies’, unavailability. (Estimated completion: March 2025) How can customers make incidents like this less impactful? • Consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/PSM0-BQ8

Read the full incident report →