Hosted Mender incident

Rate limits issue for some customers

Resolved after 2 hours 46 minutes, affecting Hosted Mender US, on November 19, 2025.

resolved Nov 19, 2025, 07:55 AM UTC

This incident has been resolved. However, a rate limit hot fix has been implemented, so we will schedule a new maintenance window soon, to apply the definitive fix.

Started
Nov 19, 2025, 05:08 AM UTC
Resolved
Nov 19, 2025, 07:55 AM UTC
Duration
2h 46m
Detected by Pingoru
Nov 19, 2025, 05:08 AM UTC

Affected components

Hosted Mender US

Update timeline

  1. investigating Nov 19, 2025, 05:08 AM UTC

    We just upgraded the hosted Mender US cluster to Kubernetes v1.33 with a blue/green approach. The new cluster (blue) right after the switch is experiencing issues with the rate limits for some customer. We are investigating the issue.

  2. identified Nov 19, 2025, 06:12 AM UTC

    We rolled back to the old green cluster, still having the same issue.

  3. monitoring Nov 19, 2025, 06:12 AM UTC

    We found the root cause, a fix has been implemented, and we're monitoring the results.

  4. resolved Nov 19, 2025, 07:55 AM UTC

    This incident has been resolved. However, a rate limit hot fix has been implemented, so we will schedule a new maintenance window soon, to apply the definitive fix.

Your product runs on more than Hosted Mender

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →