SENKO Outage History

Started Oct 21, 2026, 01:59 PM UTC· resolved Oct 21, 2026, 02:35 PM UTC

1. identified Oct 21, 2026, 01:59 PM UTC

   We are currently performing an emergency replacement of a failed NVMe drive in this node's RAID array. All customer data is safe and intact. Affected VMs will be back online as soon as the disk replacement is complete and the node has been booted back into the OS. We'll post an update once service is restored.

2. resolved Oct 21, 2026, 02:35 PM UTC

   The affected node is back online and virtual machines are now powering back on. Services should return to normal over the next 15-20 minutes as each VM completes boot.

Read the full incident report →

Started Oct 17, 2026, 06:30 AM UTC· resolved Oct 19, 2026, 03:30 PM UTC

1. identified Oct 17, 2026, 06:30 AM UTC

   The VM panel is currently under maintenance, therefore it is not available. Please contact the support team if you require any changes within the VM, such as a OS reinstall or password change. Please note that the maintenance time does not reflect the actual time it will take for the panel to come back online.

2. resolved Oct 19, 2026, 03:30 PM UTC

   Marking as resolved (see post-mortem)

Read the full incident report →

Started Aug 31, 2026, 07:25 PM UTC· resolved Sep 01, 2026, 12:22 AM UTC

1. investigating Aug 31, 2026, 05:55 PM UTC

   We are investigating the issue on the ams-9950x-odessana node in the Netherlands.

2. identified Aug 31, 2026, 06:56 PM UTC

   We found the cause of the issue and are working with the data center engineers to diagnose and resolve it as soon as possible.

3. identified Aug 31, 2026, 09:47 PM UTC

   We are continuing our work on resolving the issue.

4. resolved Sep 01, 2026, 12:22 AM UTC

   After a thorough investigation, we identified the root cause of the incident: a software issue between the AMD platform and IOMMU caused the PCI devices backing the NVMe drives on cluster node ams-9950x-odessana to malfunction on a software level, forcing the system to switch into a read-only mode and render the VMs unoperable. The issue was initially misdiagnosed as a drive failure, which led to an extended delay while we requested remote hands at the data center to replace the suspected faulty drive. Further diagnosis revealed the actual cause. We adjusted the BIOS settings accordingly, then gracefully recovered and resynced the RAID array on the node. All of the VMs are back up since 00:04 AM UTC, and all customer data is safe. However, we strongly recommend running a file system check on any virtual machines hosted on ams-9950x-odessana to rule out any latent issues. We sincerely apologize for the disruption and the extended downtime. We will be providing two extra days of service time to all customer services that were affected by this outage.

Read the full incident report →

Started Aug 27, 2026, 12:48 PM UTC· resolved Aug 27, 2026, 12:48 PM UTC

1. identified Aug 27, 2026, 11:35 AM UTC

   We are investigating the issues with the cluster nodes fra-9275f-odessana, fra-9555p-rooki, fra-9375f-budget-2, fra-9334-budget-3, ams-9575f-vulpes, ams-9575f-odessana

2. resolved Aug 27, 2026, 12:48 PM UTC

   During planned maintenance affecting five cluster nodes in our fra-9950X cluster, the data center power phase was brought down as scheduled. Due to the interaction between the motherboard firmware and the redundant power supply configuration used on these servers, the motherboards locked CPU frequency to 600 MHz. This caused severe performance degradation and VM outages for customers hosted on the affected nodes. Affected cluster nodes: fra-9275f-odessana fra-9555p-rooki fra-9375f-budget-2 fra-9334-budget-3 Separately and unrelated to the incident above, our Netherlands location experienced an issue with the VM platform that caused a partial networking malfunction, resulting in outages on: ams-9575f-vulpes ams-9575f-odessana This has already been resolved and no further action is required. ℹ️ This is an unplanned incident. No action is required from customers at this time. We will automatically provide two extra days of service to all impacted customers as compensation for this incident. We sincerely apologize for the inconvenience. This incident was not something we could have anticipated. The issue has already been resolved, and our team has restored full performance across all affected nodes. Thank you for your patience and understanding.

Read the full incident report →

Started Jun 15, 2026, 06:00 PM UTC· resolved Jun 15, 2026, 06:20 PM UTC

1. identified Jun 15, 2026, 05:25 PM UTC

   Our Finland location is currently experiencing traffic losses originating from our upstream provider's uplink at Arelion since 05:25 PM UTC. The issue has been traced to Arelion's side, with packet losses visible directly on their routers. This confirms the problem lies within their network and not our own configuration. This issue also impacts many other providers who rely on Arelion. Our upstream provider has escalated the matter, and we are currently awaiting resolution on Arelion's end. As the fix is dependent on Arelion's team, we do not have a definitive timeframe at this stage. We are staying in close contact with our upstream provider and will provide updates as soon as the situation progresses. We apologize for the disruption and appreciate your understanding. Further updates will follow as soon as we have more information.

2. resolved Jun 15, 2026, 06:20 PM UTC

   We are happy to confirm that the Finland traffic loss issue has been fully resolved as of 06:20 PM UTC. Arelion has confirmed that the problem was on their side, caused by congestion issues within their network in Finland. Our Finland network is now operating normally, and packet losses are no longer present. We will continue to monitor the situation to ensure stability remains consistent. Fun fact: this was most likely linked to the UEFA World Cup, with a live game between Belgium and Egypt taking place right now. We thank everyone for their patience and understanding throughout this incident.

Read the full incident report →

Started Jun 10, 2026, 04:40 PM UTC· resolved Jun 10, 2026, 04:40 PM UTC

1. resolved Jun 10, 2026, 04:40 PM UTC

   Over the past 2 days, our Netherlands location experienced a series of disruptions stemming from our migration to a new upstream provider. As we promised, we are providing a full post-mortem of what happened in order. What happened It all started three months ago, when our Zet.Net uplink came under a large-scale DDoS attack, heavily saturating our uplinks and putting our Netherlands location as a whole at risk. This was also heavily compounded by ongoing issues with Zet.Net themselves - their services had been severely declining, with often congestion issues during peak hours, as well as their NOC team not being very responsive. They were also unable to provide a solid solution for our network at that time. Given the attack combined with the overall deterioration in quality, we made the decision to cancel our contract and reroute all Netherlands traffic through our Germany location. While this added a few milliseconds of latency, it allowed us to protect the servers while we searched for a better long-term solution. During this period, we were actively on the lookout for a new upstream provider - one that could properly accommodate our Netherlands location and allow us to live up to the promises we've made to our customers. Once we found the right provider, we began the migration. Due to a combination of issues that arose during the process - including mistakes made on both the upstream provider's end and our own - the transition did not go as smoothly as intended. As a result, some of our traffic was unable to reach our servers properly, causing network drops and, in some cases, downtime for affected customers. Resolution We're pleased to confirm that the maintenance has already been completed, and our Netherlands network is now running on the new upstream provider. The new upstream provider features a large number of private peerings with key providers across Central and Eastern Europe, delivering even lower ping times for our customers. This upgrade also lays the groundwork for future development of our network infrastructure, including the planned deployment of a core network in one of our key data centers. Compensation We understand these issues were frustrating and caused real disruption for many of you. To make things right, we will be providing three additional days of service compensation to all impacted customers. Conclusion Looking ahead, this upgrade will allow us to further develop our tariff plans and expand our range of offerings, including custom solutions tailored to your needs. We thank everyone for their patience throughout this incident.

Read the full incident report →

Started Jun 02, 2026, 06:50 PM UTC· resolved Jun 05, 2026, 10:18 PM UTC

1. identified Jun 02, 2026, 06:50 PM UTC

   Due to an ongoing global hardware shortage, our VPS plan lineups based on the AMD EPYC 9004 series and the Ryzen 9 7950X3D CPUs in Finland, as well as Ryzen 9 9950X CPU in the Netherlands are currently fully out of stock. This shortage is driven by a worldwide surge in demand for RAM and storage, which started in the fall of last year. This surge is largely fueled by the rapid expansion of the AI industry - as major players acquire memory and drive capacity at scale, availability across the market has tightened significantly, and component pricing has risen quite a lot as a result. We want to be fully transparent about why we have chosen not to simply restock these plans at this time: at current market prices, bringing our plans back into stock would effectively force us to raise prices by at least 40% for new customers. This is something we are not willing to do. We believe in keeping our pricing fair and consistent for everyone, and we do not want new customers paying a premium driven by temporary market conditions. At this time, we are unable to provide an estimate of when our plans will be back in stock, as this very much depends on the hardware market and pricing stabilizing. We are monitoring the situation closely and will restock as soon as we are able to do so without compromising on our fair prices. We appreciate your understanding and patience during this period. Further updates will be provided as the situation develops.

2. resolved Jun 05, 2026, 10:18 PM UTC

   Following our earlier notice about the temporary shortage, both our AMD EPYC 9004 series and Ryzen 9 7950X3D VPS lineups in Finland are now back in stock - both are available in limited quantities. We've managed to bring these plans back at our usual fair pricing, without passing on the inflated component costs to our customers. Stock is limited, however, so we'd encourage anyone who's been waiting to order sooner rather than later, as availability may fluctuate. Please note that our Ryzen 9 9950X lineup in the Netherlands remains out of stock for now. We're continuing to monitor the situation closely and will provide a further update as soon as those plans can be restocked under the same pricing. Thank you for your continued patience and support. Further updates will follow as the situation develops.

Read the full incident report →

Started Apr 03, 2026, 08:40 AM UTC· resolved Apr 03, 2026, 08:40 AM UTC

1. resolved Apr 03, 2026, 08:40 AM UTC

   Following the upstream maintenance window that concluded earlier yesterday, we identified an additional incident caused by human error on the datacenter engineers side, which resulted in extended degraded performance on one affected node. During the power-related works, the engineers were reseating and replacing PSU units on the physical hosts. After replacing one of the power supply units, the server was powered on - however, the second PSU was reconnected only after the system had already fully booted into the OS. As a result of this incorrect power-on sequence, the motherboard triggered an automatic TDP cap, significantly limiting CPU power delivery. This caused severe CPU throttling and led to notable performance degradation on the affected node. This issue has now been fully resolved. The node has been rebooted under correct power conditions, the TDP limitation has been cleared, and full CPU performance has been restored. All services on the affected node are confirmed to be operating normally. Only one node (🇩🇪 FC [9555P] vulpes) was impacted by this incident. All other nodes remained fully operational and unaffected throughout. As compensation for this additional disruption, all customers hosted on the affected node will receive three extra days of service, on top of the other three days already credited for the earlier upstream maintenance. We sincerely apologize for the inconvenience. This incident was caused by an engineering oversight during the post-maintenance restoration process, and we are reviewing our procedures to prevent recurrence, as well as add more monitoring for cases like this. Thank you for your patience and understanding.

Read the full incident report →

Started Apr 02, 2026, 11:16 AM UTC· resolved Apr 02, 2026, 02:08 PM UTC

1. identified Apr 02, 2026, 11:15 AM UTC

   We have just been informed by our upstream provider of planned maintenance currently underway, which is affecting a portion of our virtual server infrastructure. As a result of this maintenance, 7 out of 69 virtual server nodes, both in Netherlands and Germany locations are currently experiencing downtime. All other nodes remain fully operational and unaffected. Estimated downtime: 30 – 45 minutes ℹ️ This is an unplanned incident. No action is required from customers at this time. We will automatically provide two extra days of service to all impacted customers as compensation for this incident. We sincerely apologize for the short notice and the inconvenience caused. The maintenance is being carried out on the upstream provider's end, and our team is actively monitoring the situation to ensure services are restored as quickly as possible. Affected customers will see their nodes return to full operation automatically once the maintenance window concludes. We appreciate your patience and understanding, and we will provide a follow-up update once all nodes are confirmed back online.

2. identified Apr 02, 2026, 11:36 AM UTC

   🇳🇱 ams-9575f-macrotis node is back online as of 11:36 AM UTC. Work is ongoing to restore full operation across the remaining nodes.

3. identified Apr 02, 2026, 12:29 PM UTC

   Unfortunately, maintenance is still being carried out. We apologize for the delay we are currently experiencing with our upstream provider.

4. identified Apr 02, 2026, 01:32 PM UTC

   VM Nodes 🇩🇪 FC [9950X] pallida, 🇳🇱 [9950X] fulva and 🇩🇪 FC [9950X] ferrilata are back online since 1:20 PM UTC

5. identified Apr 02, 2026, 01:41 PM UTC

   VM Node 🇩🇪 FC [9950X] lagopus is back online since 1:39 PM UTC. The remaining node is 🇩🇪 FC [9555P] vulpes, as well as our internal node that hosts our DNS Panel and ns1.senkodns.net.

6. resolved Apr 02, 2026, 02:08 PM UTC

   All nodes are now back online. Please allow up to 20 minutes for VMs to fully start. As compensation for the inconvenience, we're extending our original offer to three days instead of two - all affected customers will receive their compensation within today. Thank you for your patience through this incident.

Read the full incident report →

Started Mar 05, 2026, 05:16 PM UTC· resolved Mar 07, 2026, 12:15 AM UTC

1. identified Mar 05, 2026, 03:44 PM UTC

   Since 03:44 PM UTC, we have been experiencing a large-scale DDoS attack targeting customers in our Finland location. The attack is causing significant packet loss, increased latency and intermittent inaccessibility of customer services. The attack volume has exceeded 7 Tbit/s and originates from a globally distributed botnet. Our network engineering team is actively working with upstream providers to mitigate the attack. Traffic filtering and rerouting measures are currently being applied. We will continue to post updates as the situation develops. Thank you for your patience.

2. identified Mar 05, 2026, 05:50 PM UTC

   As part of our ongoing mitigation efforts, we have rerouted all traffic from our Finland location through a scrubbing PoP in Frankfurt, Germany. Active filtering is now in place and network connectivity is being restored. The attack has since escalated to 8.4 Tbit/s and is carpet-bombing our entire /24 subnet. Despite the scale, our team and upstream provider are successfully filtering the malicious traffic. As a result of the temporary reroute, you may notice higher latency than usual when accessing Finland-hosted services - this is expected behavior while traffic is being routed through Germany for scrubbing. Normal routing will be reinstated once the attack has subsided. We continue to monitor the situation in real time and will post further updates as conditions change. Thank you for your patience.

3. resolved Mar 05, 2026, 09:31 PM UTC

   Traffic has been successfully rerouted back to our Helsinki location and all services have returned to normal. The attack ceased approximately one hour ago and no further malicious traffic is being observed. We want to thank our customers for their patience throughout this incident. As a follow-up, we will be implementing automated mitigation solutions to enable faster blackholing and traffic filtering in the future - minimizing the impact of any similar events on your services. If you experience any lingering issues, please don't hesitate to reach out to our support team.

4. resolved Mar 06, 2026, 03:44 PM UTC

   We are observing a recurrence of yesterday's DDoS attack targeting our Finland location. The attack follows the same pattern as before but at a lower volume. Our upstream provider has been notified and is already actively working to mitigate the attack. We will continue to monitor the situation and provide updates as the mitigation progresses. Thank you for your patience.

5. resolved Mar 06, 2026, 04:24 PM UTC

   Our network has been stable since 03:51 PM UTC. The attack is continuing at a reduced intensity and is being actively reflected, with blackholing applied where necessary. During mitigation, our upstream provider discovered a software bug in Arista EOS on their core router - a SIGSEGV crash in the flow sampling daemon that caused the flow collector to become unstable. This most likely allowed a portion of attack traffic to slip through to the upstream network, which contributed to the network downtime our customers have experienced. An emergency upgrade has been scheduled to address this. Please be aware of the following upcoming maintenance: 7th March 2026 - 00:00 to 00:30 UTC Our upstream provider will perform an Arista EOS upgrade on their core router in Helsinki. The maintenance is expected to take up to 30 minutes, though completion may come sooner. Our Finland network will be unavailable for the duration of this window. In parallel, traffic will be once again routed through our Germany filtering PoP to reflect the ongoing attack. As a result, latency to services hosted in Finland will be elevated both before and after the maintenance window - this is expected and will normalize once the attack subsides and direct routing is restored. We apologize for the continued disruption and appreciate your understanding. Further updates will follow.

6. resolved Mar 07, 2026, 12:15 AM UTC

   The maintenance has been completed successfully. The core router of our upstream provider is back online as of 00:12 AM UTC and traffic has shifted back to Helsinki. All services should be operating normally. Thank you for your patience.

Read the full incident report →

Started Feb 22, 2026, 06:58 PM UTC· resolved Feb 23, 2026, 05:19 PM UTC

1. monitoring Feb 22, 2026, 06:58 PM UTC

   Approximately one hour ago, the fra-9575f-zerda cluster node experienced an unexpected outage due to a power rack feed issue within the data center. The interruption affected the entire cluster node. To safely restore stability, we performed a controlled power-down of the server, re-established stable power to the rack feed, and then brought the node back online. All affected virtual machines are now booting back up and should be operational going forward. Approximately 1/3 of all the VMs hosted on the node is already up. No further impact is expected at this time. We will continue monitoring the node to ensure stability. This is an unplanned incident. No action is required from customers at this time. We will automatically provide two extra days of service to all impacted customers from this incident. We apologize for any inconvenience caused.

2. monitoring Feb 23, 2026, 09:24 AM UTC

   We are aware of the elevated CPU steal time currently occurring on the fra-9575f-zerda node. Our team is actively monitoring and analyzing the situation. Prior to the power interruption and subsequent reboot, no abnormal steal time behavior was ever observed. Based on this, the issue is most likely related to a software-level condition that manifested after the restart. In parallel with our investigation, we are preparing a new cluster node to migrate a portion of the affected virtual machines within today. This will allow us to further distribute the load and mitigate the impact while we continue deeper analysis.

3. resolved Feb 23, 2026, 05:19 PM UTC

   We are pleased to confirm that the elevated CPU steal time issue on the fra-9575f-zerda cluster node has been fully resolved and proper operations have been restored. After thorough analysis, we determined that the root cause was related to how virtual machines were resuming simultaneously after the initial restart, causing excessive resource contention. To address this, we performed a full power-down and cold start of the node, and implemented changes to our virtualization configuration - specifically, VMs now start gradually with properly defined CPU quotas to prevent resource saturation during boot sequences. All virtual machines on the node are back online and operating normally. We will continue monitoring the node closely over the coming hours to ensure sustained stability. Regarding compensation: we will be providing three additional days of service to all impacted customers, on top of the two days already credited from yesterday's initial outage - totaling five extra days. We recognize this goes well beyond the actual downtime experienced and exceeds our SLA obligations, but we believe it's the right thing to do here. Our customers put their trust in us, and we understand the disruption this incident caused. This is not the first time we've gone above and beyond our policy in situations like this, and it won't be the last. We sincerely apologize for the inconvenience and thank you for your patience throughout this incident. No further action is required from customers. If you experience any remaining issues with your VM, please don't hesitate to reach out to our support team.

Read the full incident report →

Started Feb 21, 2026, 08:45 AM UTC· resolved Feb 21, 2026, 05:41 PM UTC

1. identified Feb 21, 2026, 08:45 AM UTC

   We are currently experiencing a large-scale DDoS attack targeting all VPS locations and our public web services. The attack is affecting network connectivity and may cause packet loss, increased latency, or temporary inaccessibility of services across multiple regions. Our network engineering team is actively working with upstream providers to mitigate the attack. Traffic filtering and rerouting measures are being applied to stabilize connectivity and restore full service availability as quickly as possible. We will continue to provide updates as mitigation progresses. Thank you for your patience.

2. identified Feb 21, 2026, 10:07 AM UTC

   Our Finland and Netherlands locations have been fully restored as of 09:25 UTC. The DDoS attack has ceased, and additional mitigation measures have been implemented across our network to reduce the risk of similar incidents in the future. We are currently awaiting resolution from our upstream provider in Germany. At this time, four of our routers in that location remain unavailable. This appears to be related to null-routing applied upstream, which has not yet been lifted on their end. We are in active communication with the provider to restore connectivity as soon as possible. Further updates will follow once additional information becomes available.

3. identified Feb 21, 2026, 10:23 AM UTC

   We have manually rerouted traffic to the available routers, and network connectivity has now been restored. All VPS locations and web services are operational. The earlier disruption was related to upstream null-routing in Germany following the DDoS event. While we continue coordinating with the upstream provider regarding the affected routers, traffic is currently stable and flowing normally through alternative paths. We will provide further updates once we have more information.

4. identified Feb 21, 2026, 03:41 PM UTC

   Unfortunately, DDoS attacks have resumed and are once again targeting all of our locations and public services. This may result in intermittent packet loss, increased latency, or temporary service inaccessibility across multiple regions. We are actively coordinating with our upstream providers to mitigate the attack and stabilize network connectivity as quickly as possible. Additional filtering and traffic engineering measures are being applied. Following resolution, we will be implementing enhanced automated countermeasures to improve response times and reduce the impact of similar incidents in the future. Further updates will be provided as the situation develops.

5. resolved Feb 21, 2026, 05:41 PM UTC

   As of 16:10 UTC, the DDoS attacks have stopped. Over the following two hours, we gradually rolled out routing adjustments to direct traffic back through our primary routers and restore full network capacity across all locations. All services are now operating normally. No further network issues are expected at this time. We will continue to closely monitor the situation and maintain heightened alertness to ensure ongoing stability.

Read the full incident report →

Started Feb 17, 2026, 06:58 PM UTC· resolved Feb 17, 2026, 10:56 PM UTC

1. identified Feb 17, 2026, 05:20 PM UTC

   We are currently observing intermittent network connectivity issues in our Netherlands location caused by a fault at the upstream provider ZetNet. According to the provider, their wavelength carrier is experiencing problems, with multiple transport waves currently down. This is impacting overall connectivity within the facility and may result in packet loss, increased latency, or temporary reachability issues for affected services. The upstream provider is actively working to restore the failed transport waves. We are monitoring the situation closely and will provide further updates as more information becomes available or once stability is fully restored. We apologize for the disruption and appreciate your patience.

2. resolved Feb 17, 2026, 10:56 PM UTC

   Network connectivity in our Netherlands location has been fully restored as of 10:56 PM UTC. The upstream provider ZetNet has confirmed that the issue affecting their wavelength carrier and multiple transport waves has been resolved. No further interruptions or instability are expected. The provider also noted that they are continuing internal work to improve resilience and prevent similar incidents in the future. There was no complete downtime during this incident. However, internal packet loss caused by the upstream fault resulted in intermittent connectivity degradation and brief packet loss for some services while the issue was ongoing. We will continue to monitor the location to ensure ongoing stability.

Read the full incident report →

Started Feb 11, 2026, 07:55 AM UTC· resolved Feb 11, 2026, 02:40 PM UTC

1. identified Feb 11, 2026, 07:55 AM UTC

   Since last night, we have observed major CPU performance degradation on the cluster node 🇩🇪 FC [9474F] arctic. After further investigation, the issue was traced to a failure of one of the server's redundant power supply units (PSUs), causing the system to operate at reduced performance. Due to the redundant power delivery design, the node remained online for an extended period, however, sustained high CPU load ultimately led to a crash. Service was restored via IP-KVM by temporarily lowering the CPU frequency. The cluster node is currently operational, though users hosted on this node may continue to experience reduced performance. We are actively working with the data center provider to replace the failed PSU as soon as possible and will provide further updates as the situation progresses. This is an unplanned incident. No action is required from customers at this time. We will automatically provide three extra days of service to all impacted customers from this incident, once it gets fully resolved. We apologize for the inconvenience and appreciate your patience while we resolve the issue.

2. resolved Feb 11, 2026, 02:40 PM UTC

   The issue affecting cluster node fra-9474f-arctic has been fully resolved since 8 minutes ago. The root cause was identified as a cable-related fault that prevented the second power supply unit from operating correctly. Resolution required coordination with the data center provider, which unfortunately introduced additional delay. The cluster node is now operating under normal conditions, and no further impact is expected. As a goodwill gesture, three days of additional service time have been credited to all affected customers. We sincerely apologize for the inconvenience caused once again.

Read the full incident report →

Started Feb 05, 2026, 11:39 AM UTC· resolved Feb 05, 2026, 11:39 AM UTC

1. investigating Feb 05, 2026, 09:30 AM UTC

   A part of our DE-RZ9 cluster is currently unavailable due to issues that occurred following the planned maintenance. Our team is actively working to restore full service as quickly as possible. Affected nodes: - fra-9950x-pallida - fra-9950x-zerda All customers impacted by this unplanned downtime will receive compensation in the form of three additional days added to their service period. We sincerely apologize for the inconvenience and appreciate your patience while we resolve the issue.

2. resolved Feb 05, 2026, 11:39 AM UTC

   The issue affecting a part of our DE-RZ9 cluster has now been fully resolved, and all services are operating normally again. Compensation in the form of three additional days of service will be credited automatically to all impacted customers. No action is required on your side.

Read the full incident report →

Started Feb 03, 2026, 06:54 PM UTC· resolved Feb 03, 2026, 06:55 PM UTC

1. identified Feb 03, 2026, 06:49 PM UTC

   Unfortunately, due to an unexpected issue with the node's network connectivity, 90% of the virtual servers hosted on the cluster node became unavailable. We are currently working on rebooting the VM node and resolving the issue as soon as possible.

2. resolved Feb 03, 2026, 06:55 PM UTC

   We have now resolved the issue and all affected virtual servers are coming back online.

Read the full incident report →

Started Jan 25, 2026, 02:21 PM UTC· resolved Feb 10, 2026, 02:32 PM UTC

1. identified Jan 25, 2026, 09:00 AM UTC

   Over the past several days in our Netherlands location due to a noticeable increase in overall network load, we have been observing intermittent traffic loss and a significant reduction in network speeds affecting some customer servers during peak hours. These issues are most visible in the evening, when utilization across our infrastructure reaches its highest levels. Customers may experience symptoms such as unstable latency, sporadic packet loss, temporary throughput degradation, and reduced network performance for real-time or latency-sensitive applications, including VoIP, VPNs, streaming, and other workloads that rely on consistent packet delivery over TCP. After a detailed investigation, we would like to clarify that this behavior is not related to the quality, stability, or capacity of the upstream transit provider itself. The uplink remains operational and performs as expected. Instead, the root cause is directly tied to congestion and overload occurring on our own router infrastructure in the Netherlands under high traffic conditions, primarily caused by an unexpected spike of traffic, most likely related due to the issues in our Finland location. As traffic levels have grown, the router is reaching capacity limits during peak usage, which results in packet drops and reduced performance for some connections. Since this bottleneck occurs at the routing layer, it cannot be fully resolved through local tuning alone and a capacity upgrade is required. To permanently address this issue, we have already ordered the deployment of a second uplink connected directly to our router in the Netherlands. This upgrade will significantly increase available capacity, improve redundancy, and remove the current saturation point that is causing performance degradation during high-load periods. Once completed, it will allow us to ensure stable network performance even during peak traffic hours. However, due to the specific operational processes of our upstream provider and the data center environment they utilize, this type of installation requires a longer lead time on their side. According to the timeline provided by them, the deployment may take up to an additional 10 days. This delay is caused by the need to order new cabling and arrange on-site work, including the physical presence of their engineers in the data center to perform the uplink connection. We are treating this matter with the highest priority and will continue monitoring the situation closely until the upgrade is completed. Further updates will be provided as soon as deployment progresses, and we will notify customers immediately once the additional uplink is live and the issue is fully resolved. We will also be providing service compensation to affected customers in a form of a service extension, proportional to the network disruptions. We appreciate your patience and understanding throughout this incident. Thank you for your continued trust and support!

2. identified Jan 27, 2026, 10:45 AM UTC

   The required DAC cables for the second uplink are currently in transit. As mentioned earlier, the estimated timeline for completing the upgrade remains around 10 days, with approximately 8 days remaining. We will keep you updated over the coming days as deployment progresses.

3. identified Jan 29, 2026, 06:29 PM UTC

   We are currently working on attaching the additional uplink in our Netherlands location. During this procedure, there may be a brief network interruption of up to 10 minutes while the connection is being established and bond is configured. We will continue monitoring closely and provide further updates once the upgrade is fully completed.

4. identified Jan 29, 2026, 06:45 PM UTC

   Network connectivity in our Netherlands location has been restored since four minutes ago. We are now attaching the additional uplink as part of the capacity upgrade. During this final work, there may be brief intermittent interruptions while the link is brought online and routing converges. We will will provide an update once the maintenance is over.

5. identified Jan 29, 2026, 07:16 PM UTC

   Unfortunately the data center engineers attached the DAC to the wrong server. They are currently working on correcting this issue as quickly as possible. We will provide another update as soon as the maintenance is fully completed.

6. identified Jan 29, 2026, 07:36 PM UTC

   We have brought the network back up once again to remediate the downtime. We are still currently waiting for the additional uplink to be correctly attached, and work is ongoing. Further updates will be provided as soon as progress continues.

7. identified Jan 31, 2026, 12:15 PM UTC

   The brief network interruptions observed within the last 24 hours were caused by bonding-related configuration work on the upstream provider's switch, which resulted in our links restarting while the second uplink is still not connected. We are treating this situation with the highest priority. However, the repeated errors and extended delays on the data center side are making us seriously question the operational reliability of the chosen upstream provider and their on-site team. We will provide another update immediately once we receive confirmation and the uplink is finally attached correctly.

8. monitoring Feb 02, 2026, 09:32 PM UTC

   Thanks to router-related configuration improvements made by our team, we have been able to stabilize network performance in our Netherlands location even during peak hours, while also providing better throughput and overall speeds. We are still currently waiting for the second uplink to be correctly attached, and work on the capacity upgrade remains ongoing. Further updates will be provided as soon as installation progresses.

9. resolved Feb 10, 2026, 02:32 PM UTC

   The maintenance in our Netherlands location has now been fully completed, and the second uplink has been successfully attached to our router. With the additional capacity in place, this upgrade is expected to improve overall network quality, eliminate the packet loss previously observed during peak hours, and provide more stable throughput and higher available speeds across the location. We will continue closely monitoring network performance to ensure stability and confirm the long-term effectiveness of the upgrade. Further updates will be shared if necessary.

Read the full incident report →

Started Jan 23, 2026, 04:00 AM UTC· resolved Jan 23, 2026, 04:16 AM UTC

1. investigating Jan 23, 2026, 03:50 AM UTC

   Unfortunately, due to an unexpected issue with the node's NIC, network connectivity became unavailable. As the IPMI console was also inaccessible, we had no alternative but to request Remote Hands support from the data center. The data center engineers are currently inspecting the server and will resolve the issue as soon as possible.

2. investigating Jan 23, 2026, 04:06 AM UTC

   We have now regained direct access to the node via IPMI. As the server became unavailable during the outage, a hard reboot was required to restore access. Since the server had been running continuously for a prolonged period, the motherboard has initiated RAM training. This is expected behavior on DDR5-based systems and may take up to 15-20 minutes. We will provide an update as soon as the node is back up online.

3. resolved Jan 23, 2026, 04:16 AM UTC

   The node is now back online, and the issue has been fully resolved.

Read the full incident report →

Started Jan 18, 2026, 04:51 PM UTC· resolved Jan 18, 2026, 05:18 PM UTC

1. identified Jan 18, 2026, 04:44 PM UTC

   We are currently experiencing an outage on the fra-9950x-lagopus node, which has resulted in a connectivity issue between the VM control panel and the node itself. Due to this issue, some virtual machines on this node are currently offline. Our engineers are actively investigating the root cause and working to restore normal operation as quickly as possible. Further updates will be provided as we make progress. ℹ️ This is an unplanned incident. No action is required from customers at this time. We apologize for the inconvenience and appreciate your patience while we resolve the issue.

2. identified Jan 18, 2026, 04:59 PM UTC

   We have investigated the cause of the issue affecting the fra-9950x-lagopus node. Our engineers are actively working on resolving the problem as fast as possible to restore full connectivity between the VM control panel and the node, as well as bring all affected virtual machines back online.

3. resolved Jan 18, 2026, 05:18 PM UTC

   The fix has been applied for the issue affecting the fra-9950x-lagopus node. Connectivity between the VM control panel and the node has been restored, and affected virtual machines are returning to normal operation. We are currently monitoring the situation closely to ensure full stability.

Read the full incident report →