DFDS IT incident

Unstable Kubernetes Worker Nodes

Resolved after 2 days 19 hours, affecting Kubernetes [Hellman] - Capacity/Scheduling, on March 3, 2025.

postmortem Mar 18, 2025, 08:54 AM UTC

[https://raw.githubusercontent.com/dfds/postmortems/refs/heads/master/PM2025-001 - Node system resource starvation and scheduling capacity](https://raw.githubusercontent.com/dfds/postmortems/refs/heads/master/PM2025-001%20-%20Node%20system%20resource%20starvation%20and%20scheduling%20capacity)

Started
Mar 03, 2025, 01:32 PM UTC
Resolved
Mar 06, 2025, 08:37 AM UTC
Duration
2d 19h
Detected by Pingoru
Mar 03, 2025, 01:32 PM UTC

Affected components

Kubernetes [Hellman] - Capacity/Scheduling

Update timeline

  1. investigating Mar 03, 2025, 01:32 PM UTC

    We are currently investigating an issue that is causing the odd Kubernetes worker node to become unresponsive and evict its workloads to other nodes

  2. monitoring Mar 03, 2025, 06:49 PM UTC

    We have implemented a potential fix and are monitirong the results

  3. resolved Mar 06, 2025, 08:37 AM UTC

    Nodes have been stable since the fix was applied so we consider the issue resolved

  4. postmortem Mar 18, 2025, 08:54 AM UTC

    [https://raw.githubusercontent.com/dfds/postmortems/refs/heads/master/PM2025-001 - Node system resource starvation and scheduling capacity](https://raw.githubusercontent.com/dfds/postmortems/refs/heads/master/PM2025-001%20-%20Node%20system%20resource%20starvation%20and%20scheduling%20capacity)

Your product runs on more than DFDS IT

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →