IONOS Cloud Outage History

Started Feb 04, 2026, 11:13 AM UTC· resolved Feb 04, 2026, 03:30 PM UTC

1. investigating Feb 04, 2026, 11:13 AM UTC

   There is currently an increased processing time for provisioning requests initiated via the Data Center Designer or the API. The availability and accessibility of your virtual datacenter resources will remain unaffected.

2. monitoring Feb 04, 2026, 12:43 PM UTC

   Processing time for provisioning requests has returned to normal. The service is currently being monitored

3. resolved Feb 04, 2026, 03:30 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 03, 2026, 11:38 AM UTC· resolved Feb 05, 2026, 05:49 PM UTC

1. identified Feb 03, 2026, 11:38 AM UTC

   A technical issue affecting invoice generation has been identified. We are currently implementing a resolution to restore service to the invoice endpoint. Access to these records is temporarily suspended.

2. resolved Feb 05, 2026, 05:49 PM UTC

   We have received an update from our Billing team. The API is fully operational again and we are marking this incident as resolved.

Read the full incident report →

Started Jan 30, 2026, 09:56 AM UTC· resolved Jan 30, 2026, 12:53 PM UTC

1. investigating Jan 30, 2026, 09:56 AM UTC

   We are currently investigating intermittent issues with IONOS Cloud Support Telephone Line. While mostly outgoing calls are affected, we ask customers to use [email protected] if the team cannot be reached correctly via phone.

2. monitoring Jan 30, 2026, 12:42 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jan 30, 2026, 12:53 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 29, 2026, 11:40 AM UTC· resolved Jan 29, 2026, 05:10 PM UTC

1. investigating Jan 29, 2026, 11:40 AM UTC

   We are currently investigating increased error rate in our AI Model Hub.

2. monitoring Jan 29, 2026, 01:00 PM UTC

   The team has successfully mitigated the incident. We are actively monitoring the service to ensure its full restoration.

3. resolved Jan 29, 2026, 05:10 PM UTC

   Our AI Model Hub experienced a temporary degradation, resulting in an increase in 5xx errors when interacting with the service. The issue was identified as a spike in memory usage in the API components. Our teams implemented mitigation measures to stabilize the load, and the service has been fully restored. The incident is now considered resolved.

Read the full incident report →

Started Jan 27, 2026, 04:35 PM UTC· resolved Jan 27, 2026, 06:55 PM UTC

1. investigating Jan 27, 2026, 04:35 PM UTC

   We are currently investigating alerts related to DCD and IAM service availability in TXL DC

2. investigating Jan 27, 2026, 04:37 PM UTC

   We are continuing to investigate this issue.

3. identified Jan 27, 2026, 05:08 PM UTC

   We have identified a connectivity issue preventing our identity service to reach an upstream database. Our Operations Team is currently working to mitigate the connectivity issue. We will be updating this page regularly to keep you informed about our progress.

4. identified Jan 27, 2026, 05:57 PM UTC

   We have traced the issue to an internal object storage system. Our infrastructure team is currently working on recovery. At this time, users may continue to experience DCD login issues, IAM service disruptions, and delays or errors related to provisioning. We see no evidence of further user-facing network issues and are updating the status accordingly.

5. identified Jan 27, 2026, 06:24 PM UTC

   We have recovered the affected Object Storage and are currently monitoring the recovery of affected systems and services.

6. monitoring Jan 27, 2026, 06:24 PM UTC

   We have recovered the affected Object Storage and are currently monitoring the recovery of affected systems and services.

7. resolved Jan 27, 2026, 06:55 PM UTC

   We are marking this incident as resolved. We will compile a Root Cause Analysis and will publish it as an update to this incident.

8. postmortem Feb 02, 2026, 04:05 PM UTC

   We have finished our research into this incident and want so share the following Root Cause Analysis: **What happened?** On January 27, 2026, a storage incident occurred in our Berlin region, leading to I/O performance degradation. This impacted the availability of management and customer-facing services. **How did this happen? \(Technical Root Cause\)** The incident was triggered during a scheduled maintenance window involving updates to parts of our supporting infrastructure.. * _Initialization Failure_: Following a node reboot, several storage daemons failed to reconnect due to a configuration mismatch stemming from a previous hardware replacement. * _Operational Error_: During the troubleshooting process, a manual command was executed to clear stale storage entries. This command inadvertently removed active storage components from other nodes while the cluster was already in a vulnerable, degraded state. * _Performance Impact_: The loss of these additional components resulted in the temporary unavailability of specific data segments. This caused a "hang" in I/O operations for some management services relying on that storage, leading to timeouts and service interruptions. Unexpectedly, two failover instances did not behave nominally, which led to the short-term impact on the IAM \(auth management\) system. Full functionality was restored once engineers recovered the missing storage components and initiated a cluster-wide data rebalancing. At no point systems that are handling or storing customer data were directly involved in this incident, however, our IAM services were affected by the performance impact of the incident, leading to customer facing service outage for the DCD login. **What are we doing to prevent this from happening again?** To increase the resilience of our internal supporting infrastructure and minimize the impact of human error, we are reviewing and strengthening the following measures: * _Refinement of Hardware Procedures_: We are updating our disk replacement and decommissioning workflows to include more robust verification steps, ensuring hardware identifiers remain consistent through system reboots. * _Guardrails for Management Commands_: We are implementing software safeguards and updated standard operating procedures \(SOPs\) to restrict high-impact cluster commands while a storage environment is already showing signs of degradation. * _Redundancy Improvements:_ We are conducting a review of services that experienced failover issues to ensure that a local disruption does not impact global service availability.

Read the full incident report →

Started Jan 23, 2026, 09:49 PM UTC· resolved Jan 24, 2026, 01:52 AM UTC

1. investigating Jan 23, 2026, 09:49 PM UTC

   We have registered an increase in error rate for our S3 service in TXL

2. monitoring Jan 23, 2026, 11:08 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jan 24, 2026, 01:52 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 17, 2026, 07:13 PM UTC· resolved Jan 18, 2026, 10:11 PM UTC

1. investigating Jan 17, 2026, 07:13 PM UTC

   Summary: We have identified a physical hardware failure within our firewall cluster at the Las Vegas data center. Impact: All provisioning jobs for new and existing Cloud instances in this region are currently failing. Please note that existing, active instances are not affected by this issue and operate normally. Current Status: Our team is on-site installing replacement hardware. We anticipate a delay in full restoration and will provide an updated ETA as soon as possible.

2. monitoring Jan 17, 2026, 10:48 PM UTC

   A fix has been implemented on the affected physical device and normal provisioning services are back to running.

3. resolved Jan 18, 2026, 10:11 PM UTC

   After extensive monitoring of final fix, continued stability has been restored

Read the full incident report →

Started Jan 15, 2026, 08:21 AM UTC· resolved Jan 15, 2026, 09:25 AM UTC

1. investigating Jan 15, 2026, 08:21 AM UTC

   We are writing to inform you that network services for VMware-based Private Clouds in Berlin are currently impacted. This is affecting both management VPN access as well as traffic to / from individual VMs. Network technicians began working on this soon as the outages were detected, and will isolate the problem and solve the issue as quick as possible. We will inform you as soon as the functionality has been restored.

2. monitoring Jan 15, 2026, 08:37 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jan 15, 2026, 09:25 AM UTC

   This incident has been resolved.

4. postmortem Jan 26, 2026, 11:06 PM UTC

   **What happened?** On 15.01.2026, we experienced a network incident that resulted in a loss of connectivity to key management devices in Berlin, as well as a disruption of Private Cloud management and production traffic. **How did this happen? \(Technical Root Cause\)** The incident occurred during a planned maintenance window intended to introduce a new VLAN to the management devices. Human error during command execution led to unexpected behavior: instead of appending the new VLAN, the existing VLANs on the trunk between the core and management devices were overwritten. Consequently, only the new VLAN remained active. The removal of these VLANs led to the following cascading outages: * Management Loss: Loss of access to multiple management interfaces in the Berlin region. * Storage Disruption: The affected management devices hosted interfaces for internal storage clusters. When the associated VLANs became unreachable, the storage clusters were disconnected. During the final stages of recovery, restarting the affected services led to a brief secondary outage. * Data Integrity Protection: This disconnection forced several VMs and Edge Nodes into read-only mode to prevent data corruption. * Routing Issues: Since the NSX Edge Nodes \(which bridge virtual networks with the physical environment\) were impacted, external routing was lost. This also resulted in a loss of connectivity between the virtual machines and the internet. **What are we doing to prevent this from happening again?** To improve network resilience and prevent a recurrence, we have identified the following action items: * Architecture Review: We will conduct a comprehensive audit of the management device architecture to ensure better segmentation. \(Target: Q3 2026\) * VLAN Configuration Mitigation: We will evaluate and implement solutions to prevent unintended VLAN changes, such as improved configuration safeguards or automation scripts. \(Target: Q1 2026\)

Read the full incident report →

Started Jan 13, 2026, 03:37 PM UTC· resolved Jan 13, 2026, 05:16 PM UTC

1. investigating Jan 13, 2026, 03:37 PM UTC

   The network team is currently investigating an issue with IPV6 addressing in TXL

2. resolved Jan 13, 2026, 05:16 PM UTC

   The Degradation In IPV6 Addressing is now resolved.

Read the full incident report →

Started Jan 09, 2026, 03:42 PM UTC· resolved Jan 10, 2026, 11:55 AM UTC

1. investigating Jan 09, 2026, 03:42 PM UTC

   We are currently investigating increased error rate in our TXL site

2. monitoring Jan 09, 2026, 04:41 PM UTC

   We have traced the increased error rate to a node that ran into problems during an OS update and is being repaired. We are downgrading the impact and will resolve the incident once the work is completed.

3. resolved Jan 10, 2026, 11:55 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Dec 22, 2025, 01:54 AM UTC· resolved Dec 22, 2025, 11:22 AM UTC

1. investigating Dec 22, 2025, 01:54 AM UTC

   The S3 team is currently working on an issue impacting S3 in Frankfurt. Based on the last reports, we are seeing High error rates likely caused by communication issues. we will update regularly as the issue is investigated and resolved.

2. investigating Dec 22, 2025, 02:22 AM UTC

   Early indications point to an internal network issue, impacting communication between Buckets. External traffic is not impacted, and we do not see other customer-facing resources affected either.

3. identified Dec 22, 2025, 02:42 AM UTC

   The team has identified the source of the issue and has begun troubleshooting and remediation

4. identified Dec 22, 2025, 02:54 AM UTC

   We are continuing to work on a fix for this issue.

5. monitoring Dec 22, 2025, 03:16 AM UTC

   The issue has been resolved, and services should be returning to normal. The team is continuing to monitor the system recovery. We will publish the RCA as soon as the team has had a chance to review the records, and determine a cause

6. monitoring Dec 22, 2025, 03:18 AM UTC

   We are continuing to monitor for any further issues.

7. monitoring Dec 22, 2025, 04:29 AM UTC

   The team is continuing to run repairs and monitor the situation. Some customers may notice some degraded service as things work themselves out and all systems return to normal. Customers may also notice that Backups and so on did not run overnight. As mentioned, the RCA will follow along with more information.

8. resolved Dec 22, 2025, 11:22 AM UTC

   This incident has been resolved.

9. postmortem Dec 23, 2025, 08:40 AM UTC

   **What happened?** We experienced a network incident resulting in connectivity issues between our FRA and FRE data centers. This led to a high error rate for services hosted on a segment of S3 hosts in the FRA region. **How could this happen? \(Technical Root Cause\)** The incident was caused by a software failure within the Border Gateway Protocol \(BGP\) control plane on a specific network switch. This failure caused a Layer 3 routing failure, which resulted in outgoing traffic from a segment of S3 hosts being blackholed \(unexpectedly dropped\). A network redundancy mechanism designed to prevent traffic blackholing in failure scenarios did not activate because the physical network connections remained operational, even though the critical control plane service had failed. Since the physical links did not go down, the redundancy mechanism was bypassed. The issue was ultimately resolved by rebooting the affected network switch. **What are we doing to prevent this from happening again?** To prevent a recurrence and improve network stability, we are implementing the following measures: * Improved Monitoring: Enhancing BGP and container monitoring on all network devices to detect control plane failures faster. \(01. 2026\) * Vendor Support: Engaging with our network switch supplier through an open vendor support case to address the underlying software issue. \(ONGOING\) * Resiliency Investigation: Investigating and implementing improvements to enhance the overall resiliency of our network fabric devices and efficacy of failover mechanisms and triggers. \(ONGOING\)

Read the full incident report →

Started Dec 18, 2025, 04:51 PM UTC· resolved Dec 18, 2025, 07:31 PM UTC

1. investigating Dec 18, 2025, 04:51 PM UTC

   We are currently investigating degraded performance in our Kubernetes service.

2. identified Dec 18, 2025, 05:03 PM UTC

   We have traced the issue back to a specific control plane cluster. Customers using the affected control plane cluster will currently be unable to deploy changes. Our Kubernetes Team is working towards restoring the affected cluster.

3. monitoring Dec 18, 2025, 05:20 PM UTC

   We have restored the affected control plane cluster. We are now restarting related API servers and then monitoring the system for any anomalies before marking the incident as resolved.

4. resolved Dec 18, 2025, 07:31 PM UTC

   We are marking this incident as resolved. All services behave normally.

Read the full incident report →

Started Dec 18, 2025, 02:30 PM UTC· resolved Dec 18, 2025, 03:53 PM UTC

1. investigating Dec 18, 2025, 02:30 PM UTC

   We are currently investigating connectivity issues in our TXL and FRA2 datacenter.

2. identified Dec 18, 2025, 03:08 PM UTC

   We are currently investigating a service degradation in our internal monitoring setup in TXL. We do not currently see evidence for customer facing issues in TXL, so we are making TXL as operational again for now. We are currently investigating a potential DNS issue in FRA2 and will keep you informed about it here.

3. monitoring Dec 18, 2025, 03:28 PM UTC

   We are marking FRA2 as operational. Services that were marked as affected (IAM) report full availability again. We are investigating the incident as two separate incidents that happened at the same time. A internal monitoring system degradation in TXL with no customer facing impact, as well as a suspected DNS issue in FRA2 that temporarily affected IAM services. We consider both incidents resolved and are monitoring all involved systems for anomalies.

4. resolved Dec 18, 2025, 03:53 PM UTC

   We are marking this incident as resolved. We will share a root cause analysis as soon as it is compiled.

Read the full incident report →

Started Dec 17, 2025, 11:26 AM UTC· resolved Dec 17, 2025, 05:33 PM UTC

1. investigating Dec 17, 2025, 11:26 AM UTC

   We are currently investigating a service degradation for our compute service in TXL

2. monitoring Dec 17, 2025, 11:38 AM UTC

   We have received note from our compute team that the issue has been mitigated. We are currently monitoring the situation while the investigation is ongoing.

3. resolved Dec 17, 2025, 05:33 PM UTC

   We are marking the incident as resolved. We will provide an RCA once it is compiled.

Read the full incident report →

Started Dec 15, 2025, 09:13 PM UTC· resolved Dec 16, 2025, 04:01 AM UTC

1. investigating Dec 15, 2025, 09:13 PM UTC

   We are currently investigating a network issue in our TXL data center.

2. monitoring Dec 15, 2025, 10:32 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Dec 16, 2025, 04:01 AM UTC

   Update: 16.12. 07:30 - We are currently waiting for the Root Cause Analysis and will publish it here. This incident has been resolved.

4. postmortem Dec 18, 2025, 11:15 AM UTC

   **What happened?** During a planned infrastructure maintenance window \([https://status.ionos.cloud/incidents/rh8l4xmdpdk8](https://status.ionos.cloud/incidents/rh8l4xmdpdk8)\) which included a network control plane upgrade and a provisioning software update, two distinct issues led to service disruption for a number of Virtual Machines \(VMs\) across multiple compute nodes. * Issue 1 \(Configuration Error\): A misconfiguration on one of our compute nodes caused the network control plane service to fail. This resulted in a loss of connectivity for VMs situated on the node. * Issue 2 \(Software Bug\): A bug in our provisioning software was discovered during a rollout, causing firewall configuration changes to fail on two other nodes. This left some customer network configurations in a stuck state. **How was this possible \(technical root cause\)?** This particular maintenance was rolled out in multiple datacenters already without issues. In TXL the following circumstances lead to the incident: * Configuration Management Error: A software package installation done prior to the maintenance resulted in an unexpected update to a dependency on a specific compute node. This prevented the new version of our core networking software from starting correctly after its deployment. * Provisioning Software Defect: An update to our provisioning system introduced a software defect \(inconsistent VNIC entries in the SDN database\) that prevented necessary firewall configuration changes from completing successfully, leading to an inconsistency in the network configuration state for the affected virtual private data centers \(VDCs\). **What will we do to prevent this from happening again?** We are committed to preventing recurrence through the following measures that will ensure that we are able to react better to unexpected configurations during maintenance work: * Enhanced Rollout Procedures: The issue was spotted with a delay after the maintenance was conducted. We are further improving our update processes by adding more comprehensive pre- and post-maintenance checks, to actively confirm the desired state after the maintenance is finished and prevent confusion \(expected vs. unexpected alerts\) in alerting and monitoring teams during and after maintenance \(DONE\) * Resource Alignment: We will ensure that specialized network development engineers are on standby during all future core network control plane upgrades to address issues immediately. \(DONE\) * Change Synchronization: We will align and prevent updates on the same day to core network and virtual private data center provisioning systems to minimize the risk of complex failure patterns. \(DONE\) * Software Fix: Our development team is actively working on a hotfix for the identified bug in the provisioning software. \(QA Ongoing, Release expected this week\) * Incident Classification and Communication: We are simplifying the classification of incidents and automate the creation of initial Status Page updates to ensure that high impact incidents are classified as such early and are published to stakeholders quicker. \(Within Q1 2026\) We are currently rolling out a series of improvements to our network setup in all our datacenters. These upgrades will bring improved stability and performance to all our customers and partners. This incident is connected with this work and we understand that this has caused disruption. We believe that the measures we have taken now will make upcoming rollouts safer while allowing us to continue executing on our improvement roadmap with both speed and diligence.

Read the full incident report →

Started Dec 12, 2025, 07:27 PM UTC· resolved Dec 12, 2025, 10:43 PM UTC

1. investigating Dec 12, 2025, 07:27 PM UTC

   We are currently investigating increased error count in our AI Model Hub.

2. investigating Dec 12, 2025, 08:09 PM UTC

   Error rate has increased. Our Model Hub Team is working towards restoring normal operations.

3. identified Dec 12, 2025, 08:24 PM UTC

   The team has identified the issue affecting the {{baseUrl}}/v1/models endpoint and is working towards a resolution. Other Hub Endpoints should be unaffected.

4. monitoring Dec 12, 2025, 10:15 PM UTC

   The team has successfully mitigated the incident. We are currently monitoring latency to ensure that the service is fully restored.

5. resolved Dec 12, 2025, 10:43 PM UTC

   We are marking this incident as resolved. We will publish a RCA as soon as it is compiled.

Read the full incident report →

Started Dec 03, 2025, 09:37 PM UTC· resolved Dec 04, 2025, 12:50 AM UTC

1. investigating Dec 03, 2025, 09:37 PM UTC

   We are currently investigating increased error rates for our Object Storage in FRA

2. identified Dec 03, 2025, 10:11 PM UTC

   We have identified a performance issue regarding the metadata engine. The team is currently evaluating mitigation strategies to reduce the error count while maintaining system stability.

3. resolved Dec 04, 2025, 12:50 AM UTC

   Error Rate is back to baseline. We are working on permanent performance improvements

Read the full incident report →

Started Nov 27, 2025, 08:08 PM UTC· resolved Nov 28, 2025, 10:09 AM UTC

1. investigating Nov 27, 2025, 08:08 PM UTC

   We are currently investigating increased error count in two datacenters.

2. identified Nov 27, 2025, 08:39 PM UTC

   Our Storage Team is currently working towards stabilizing the service in both data centers. Error rate in TXL has dropped.

3. monitoring Nov 27, 2025, 09:30 PM UTC

   The Object Storage team has identified a likely culprit for the increased error rate and has restarted the service. We are currently monitoring the error rate in both data centers

4. resolved Nov 28, 2025, 10:09 AM UTC

   We consider the incident resolved. The error rate has stayed at baseline levels over the last few hours. We will publish a Root Cause Analysis once it is completed.

Read the full incident report →

Started Nov 27, 2025, 11:11 AM UTC· resolved Nov 27, 2025, 02:23 PM UTC

1. investigating Nov 27, 2025, 11:11 AM UTC

   We are currently investigating an increase in error rates for our S3 Service in FRA

2. identified Nov 27, 2025, 11:33 AM UTC

   Issue traced to a faulty S3 node. Mitigation is currently in progress.

3. monitoring Nov 27, 2025, 11:50 AM UTC

   We are seeing dropping error rates and believe that the migration worked. We will monitor the situation to ensure stable service.

4. resolved Nov 27, 2025, 02:23 PM UTC

   This incident has been resolved. We will publish a Root Cause Analysis (RCA) once our internal investigation is complete.

Read the full incident report →

Started Nov 24, 2025, 11:52 AM UTC· resolved Nov 24, 2025, 06:35 PM UTC

1. investigating Nov 24, 2025, 11:52 AM UTC

   We are investigating a potential issue in our S3 Object Storage

2. monitoring Nov 24, 2025, 12:02 PM UTC

   We identified a single node experiencing degraded performance caused by a disk hardware issue. Mitigation steps have been applied. We are currently monitoring the environment to ensure all performance metrics return to expected levels.

3. resolved Nov 24, 2025, 06:35 PM UTC

   Error rates remained low and are now below baseline. We mark this incident resolved. While the root cause - a disk failure - was addressed quickly, high concurrent traffic delayed the return to baseline error rates. We maintained an extended monitoring period to ensure full stability. We are also confident that our upcoming S3 service improvements will significantly mitigate prolonged recovery times in similar future incidents.

Read the full incident report →

Started Nov 19, 2025, 07:10 PM UTC· resolved Nov 19, 2025, 07:18 PM UTC

1. investigating Nov 19, 2025, 07:10 PM UTC

   Our ES/VIT location is currently experiencing a power outage. We will keep you updated.

2. investigating Nov 19, 2025, 07:15 PM UTC

   Our infrastructure team in Spain has reported a power outage. We are currently validating whether there is customer facing impact. We will publish another update at 19:45 UTC latest, or as soon as new information is available.

3. resolved Nov 19, 2025, 07:18 PM UTC

   We received note from the responding team that there is no customer impact. All components operate normally. We are resolving this incident.

Read the full incident report →

Started Nov 18, 2025, 07:54 AM UTC· resolved Nov 18, 2025, 02:09 PM UTC

1. investigating Nov 18, 2025, 07:54 AM UTC

   We are currently investigating further degradations in the performance. This may be related to an incident earlier in the night. We will provide all of our customers with regular updates.

2. identified Nov 18, 2025, 08:58 AM UTC

   Our team is currently assessing the situation. While most customers are seeing performance return to nominal levels, we are still seeing some sporadic performance degradation. We will continue to update all of our customers as our teams work to resolved all issues as fast as possible

3. monitoring Nov 18, 2025, 10:59 AM UTC

   Performance should be back to normal for all or most customers. Our team is currently working on an RCA, which we will publish here as soon as that is available.

4. resolved Nov 18, 2025, 02:09 PM UTC

   All services are reporting normal. RCA to follow.

Read the full incident report →

Started Nov 17, 2025, 10:55 PM UTC· resolved Nov 18, 2025, 10:33 PM UTC

1. investigating Nov 17, 2025, 10:55 PM UTC

   We are currently investigating a storage issue

2. identified Nov 17, 2025, 11:14 PM UTC

   We are currently recovering a storage server outage in the datacenter. We are currently migrating affected storages/VMs from the host. We will provide another status update before 23:30 UTC.

3. identified Nov 17, 2025, 11:27 PM UTC

   We are continuing to work on a fix for this issue.

4. identified Nov 17, 2025, 11:31 PM UTC

   We needed to update the scope of the incident. A recovery plan has been put together, and recovery efforts are currently already underway. We are monitoring the process closely and will share more details as they become available. Another update will be made until 00:00 UTC latest.

5. identified Nov 17, 2025, 11:44 PM UTC

   As recovery efforts progress, we are beginning to see improvements in service availability. While we are still actively mitigating the incident, we want to share a preliminary, high-level overview of the situation: We have identified a machine that caused a widespread network disruption in FRA. This disruption cascaded, degrading the performance of a network that provided connectivity to a storage server. The resulting loss of connectivity led to service disruptions and degradation for compute resources in the datacenter. We will keep you updated about the progress of the recovery as well as insights into the root cause of the incident.

6. identified Nov 18, 2025, 12:11 AM UTC

   The recovery of the affected volumes has progressed well overall. We have a subset of volumes were recovery takes longer than expected. We are currently looking into restoring the remaining volumes and will then restore redundancy.

7. identified Nov 18, 2025, 12:27 AM UTC

   We believe we have identified a solution for the volumes that are still unavailable (approx.. 10%) and are currently rolling out the fix to them. In parallel, we are working to restore redundancy to the already-recovered volumes and are making good progress in this area, as well.

8. monitoring Nov 18, 2025, 01:01 AM UTC

   All the remaining volumes have been successfully recovered and redundancy is being established to the last volumes, as well. We also have seen services recover. We are closely monitoring the situation and are completing final tasks to close the incident. We have also started investigation into the root cause of the incident and will publish the analysis here. We want to thank you for your patience thus far and commit to a transparent and thorough investigation.

9. monitoring Nov 18, 2025, 01:43 AM UTC

   We want to share another note on the recovery efforts. While at the infrastructure level the incident is resolved, we recommend checking the status of virtual machines and resources deployed in your environment. Depending on how the operating systems and/or applications have been able to deal with the loss of connectivity to the storage, crashes and freezes may have occurred that might require manual intervention (e.g., a restart) to resolve. Unfortunately, we cannot perform this action for you and want to make you aware of this important check to make sure your environment is fully operational after this incident. The team is currently working on compiling the Root Cause Analysis. We expect that we will have a preliminary version to share with you today.

10. resolved Nov 18, 2025, 10:33 PM UTC

    After monitoring for and responding to residual issues, we are making this incident as resolved and will publish our preliminary Root Cause Analysis.

11. postmortem Nov 18, 2025, 10:33 PM UTC

    _This is our Root Cause Analysis. We will update the RCA as we complete our investigation, inventory, and planning of repair items._ _Update 19.11.2025 - Expanding on RTRS connection loss as contributing factor_ _Update 11.12.2025 - Adding repair items related to network upgrades and their ETAs_ **What Happened?** A defective Host Channel Adapter \(HCA\) caused instability in the InfiniBand fabric starting at 2025-11-17 22:08:37 UTC. While this initial disruption was relatively brief, these instabilities \(specific, frequent, simultaneous reconnect attempts over a short period of time\) severed storage connections and triggered a bug in a software package used on our storage servers, causing an exception and triggering reboots across several storage servers at the same time. The loss of connectivity combined with these reboots caused loss of storage connectivity across VMs in the affected cluster. **How was that possible?** While the initial network disruption was relatively brief, it severed RTRS \(RDMA \(Remote Direct Memory Access\) Transport\) connections to some storage servers. It also triggered a bug in a software package used in our storage servers. This caused several storage servers to reboot unexpectedly, which decreased redundancy and increased the likelihood of volumes becoming unavailable to VMs. The HCA hardware fault, combined with the severed RTRS sessions and the unexpected reboot of several storage servers, led to a loss of redundancy and connectivity and has been identified as the technical root cause of this incident. While the infrastructure could be recovered relatively quickly, recovering the affected VMs and services took longer than expected. We are currently investigating the exact reason for the extended recovery duration. **How did we respond?** The InfiniBand ports of the problematic server were shut down, successfully isolating the defective HCA. The fabric topologies reconverged quickly. The resulting storage connectivity issues needed to be addressed by re-attaching storage, then re-establishing redundancy for the storage, and reprovisioning/resetting affected VMs. **What are we doing to prevent this from happening again?** While the investigation is still ongoing, we have already inventoried the following actions. We will assign ETAs to them as soon as investigation and planning is finished. * Hardware & Fabric Resilience * HCA Replacement: Replace the defective HCA and thoroughly test the new component. \(Done\) * InfiniBand Migration: \(End of February 2026\) * Control Plane Hardening and SDN/VNF Updates \(ongoing - End of December 2025\) * Storage Stability * Bug Mitigation: Investigate and fix the software package bug to prevent storage server reboots triggered by similar incident. \(Done - In Rollout\) * Resilience Review: Implement measures to prevent this specific cascading failure in the future. \(Ongoing\) * Improved Recovery * Improve tooling and procedures to allow for a swifter recovery, specifically of storage connectivity loss at scale. \(Ongoing\) As we continue our investigation, we will update and add to this Root Cause Analysis. We would like to thank your for your patience and teamwork over the last hours.

Read the full incident report →

Started Nov 17, 2025, 01:59 PM UTC· resolved Nov 17, 2025, 07:20 PM UTC

1. investigating Nov 17, 2025, 01:59 PM UTC

   We are currently investigating high latency for S3 Object Storage

2. investigating Nov 17, 2025, 02:25 PM UTC

   We are currently investigating error rates above baseline in our S3 Object storage in our TXL (eu-central-3) location. Since the start of the incident, the error rate has dropped, but is still above threshold. We will update the status page every 30 minutes or when new important information becomes available. You can expect the next update at 15:00 UTC. Thank you for your patience as the team works towards a quick resolution.

3. identified Nov 17, 2025, 02:59 PM UTC

   We have identified abnormal load levels for some buckets and are currently deploying mitigations to reduce the impact further.

4. monitoring Nov 17, 2025, 03:17 PM UTC

   We have deployed mitigations and are currently monitoring the development of latency and error rate.

5. resolved Nov 17, 2025, 07:20 PM UTC

   The performance and latency have been at acceptable levels. We will mark this incident as resolved. We will provide a Root Cause Analysis here once the investigation is completed.

Read the full incident report →

Started Nov 14, 2025, 02:21 PM UTC· resolved Nov 14, 2025, 04:13 PM UTC

1. investigating Nov 14, 2025, 02:21 PM UTC

   We are currently investigating reports related to connectivity issues in our TXL datacenter. We will update you as soon as possible.

2. investigating Nov 14, 2025, 02:33 PM UTC

   We are currently investigating instances of Border Gateway Protocol session flapping in the TXL datacenter. Our network experts are currently pinpointing the root cause of the issue. You can expect the next update here at 15:00 UTC.

3. identified Nov 14, 2025, 02:56 PM UTC

   We have identified and disabled a defective switch that we believe was a potential culprit. We are now closely monitoring the environment to assess the impact of this action, as we suspect there may be additional contributing factors. Our investigation continues, and we will provide the next update by 15:30 UTC at the latest.

4. identified Nov 14, 2025, 03:02 PM UTC

   We are observing some improvements in our monitoring metrics after our recent intervention, and some previously affected systems are reporting back as healthy. We will continue to monitor the environment closely as we look for any other potential contributing factors. We will provide our next update no later than 15:30 UTC

5. monitoring Nov 14, 2025, 03:28 PM UTC

   We are seeing further improvement across the environment. We are currently actively monitoring the recovery and are confirming the Root Cause of the Incident and will publish the result here.

6. resolved Nov 14, 2025, 04:13 PM UTC

   All pending systems have reported in as healthy. We will mark this incident as resolved. We will publish the Root Cause Analysis (RCA) here as soon as it becomes available. Thank you for your patience.

7. postmortem Nov 15, 2025, 01:17 PM UTC

   _Update 18.11.2025 - After further investigation we want to update our preliminary Root Cause Analysis with additional insights_ _Update 24.11.2025 - We want to provide more concrete timelines for the action items_ **What happened:** On 14.11.2025 between 13:54 and 14:47 UTC we observed a network connectivity issue in our TXL datacenter, affecting multiple services and leading to increased latency and packet loss. **This happened because:** A datacenter cluster experienced network instability due to a broken failover behavior in the control plane of our InfiniBand \(IB\) fabrics. The core issue was a constant switching of Master/Standby roles between the redundant InfiniBand Subnet Managers, leading to inconsistencies in the fabric. The root cause of this instability was identified as overly aggressive timeout and retry settings in the Subnet Manager’s High Availability \(HA\) configuration. These sensitive settings allowed a minor, otherwise insignificant latency spike to trigger an initial failover. This created a self-perpetuating cycle: the resource overhead from the initial failover caused a spike in CPU load on the IB gateways. This increased load resulted in further latency, which the aggressive HA settings interpreted as a failure, immediately triggering another failover. This loop continued until the Subnet Manager election process stabilized, allowing the system to recover automatically. **What we are doing to prevent this from happening again:** 1. Further investigation into the Subnet Manager Log Messages to identify the trigger of the unexpected switching behavior \(completed\) 2. Hardening and improvement of the Subnet Manager High Availability \(HA\) configuration \(first sites within Q4\) 3. Migration of Subnet Manager to separate servers that are not part of the data plane \(first sites within Q4\) We understand that the incident has had a negative impact on our customers and partners. We believe that these steps will make our setup more resilient and help avoid a similar incident in the future.

Read the full incident report →