Is CloudAMQP down?

Between 19:42 UTC on 29 October and 05:04 UTC on 30 October, broker and queue level metrics were not delivered to legacy third-party metrics integrations (Datadog, CloudWatch, New Relic, Stackdriver, Splunk, Librato) for a subset of clusters. Server metrics were not affected. Prometheus-based (v3) metrics and integrations were not affected. The cause was a stalled process in our legacy metrics collection pipeline. It has been restarted and delivery has resumed. Metrics from the affected period cannot be backfilled. We are adding safeguards so this failure is detected and recovered from automatically.

We're investigating connectivity issues affecting some servers in Azure West Europe. We'll post an update as soon as we know more.

Connectivity to servers in southeast Asia has been restored.

We're investigating connectivity issues affecting some servers in Google europe-west2. We'll post an update as soon as we know more.

We are continuing to investigate this issue.

Incident is limited to servers in europe-west2-b

GCE is still reporting issues, but we are seeing some API calls succeed.

GCE has confirmed that a power outage in europe-west2-b caused the disruption. The power issue has been mitigated, and full recovery of affected workloads is tentatively expected by 13:00 UTC today. We continue to monitor the recovery.

We are continuing to monitor for any further issues.

This incident has been resolved.

We're investigating connectivity issues affecting some servers in Azure West Europe. We'll post an update as soon as we know more.

Azure is aware of this issue and are actively investigating. We'll post another update as soon as we know more.

Azure has identified this issue. They have narrowed the issue to storage accessibility affecting the region. Their teams is actively performing recovery actions on the affected infrastructure while continuing to investigate the underlying cause. We are monitoring recovery progress and working to restore affected resources.

We have been able to establish communication with instances in Azure West Europe. Majority of the instances are back up and running. Azure still has some VMs to resolve.

All CloudAMQP instances are up and running in Azure West Europe.

**Summary** CloudAMQP lost connectivity with a subset of clusters in Azure west Europe availability zone 2 on October 9th. Single node clusters in AZ2 were most impacted by the incident. ‌ **Impact** Customers were unable to access data or mange nodes impacted during the incident. Timeline of events from Azure. * 12:25 UTC on 09 October 2026 – We identified this issue via our internal service monitoring alerts. * 12:29 UTC on 09 October 2026 – Customer impact began, with VM disk failures detected in the affected West Europe storage scale unit. * 2:30 UTC on 09 October 2026 – The investigation identified loss of heartbeat across multiple storage nodes and fault domains, resulting in the storage scale unit to go in distress mode. * 15:32 UTC on 09 October 2026 - We took the storage scale unit out of distress mode, isolating the disks on the impacted storage servers to enable recovery of the remaining disks. At this stage, most of the affected customers began seeing recovery. * 21:00 UTC on 09 October 2026 - Datacenter and Storage engineers continued executing the recovery plan on multiple storage nodes. * 23:27 UTC on 09 October 2026 – Efforts to recover the affected storage nodes were successful, and work to recover the virtual disks continued. Subsequent recovery actions included node-level recovery and restoration of affected storage resources. * 03:00 UTC on 10 October 2026 – Service functionality recovered, and customer impact was mitigated. **Root cause** Azure experienced a platform issue resulted in an impact to Azure Virtual Machines storage disks in the West Europe region AZ 2. Impacted customers may have experienced connectivity failures, unexpected restarts, or Virtual Machines entering a failed state in the region. ‌ **Resolution** Machine access was restored by Azure in about 3 hours. CloudAMQP was able to restore communication with the impacted nodes. We continued to monitor updates from Azure to ensure nodes were no longer impacted.

We are seeing degraded network connectivity between some non-Azure networks and Azure UAE North (azure-arm::uaenorth), caused by a routing problem at an upstream transit provider. All servers in the region are up and processing messages, and monitoring, metrics and logs are working as normal. Customers whose clients connect from networks affected by this routing problem may also see connection timeouts. We have reported the problem to the provider and are routing our own management traffic around it.

Routing is back to normal.

This incident has been resolved.