Resolved on September 30, 2025.
On September 25th, we updated the internal Redis Cluster to a new provisioning model, but it turned out that this new deployment was applied without a proper Kubernetes Priority Class being set. So we found out that during a new regular hosted Mender deployment, the pods with higher priority and "preempt = true" could preempt Redis pods to be scheduled faster. We applied a fix on the 30th of September, but during the previous 5 days we had at least four Redis Cluster issues, which lasted 1-2 minutes each. We apologize for the issues this incident might have caused to you.
Update timeline
- resolved Sep 30, 2025, 01:30 PM UTC
On September 25th, we updated the internal Redis Cluster to a new provisioning model, but it turned out that this new deployment was applied without a proper Kubernetes Priority Class being set. So we found out that during a new regular hosted Mender deployment, the pods with higher priority and "preempt = true" could preempt Redis pods to be scheduled faster. We applied a fix on the 30th of September, but during the previous 5 days we had at least four Redis Cluster issues, which lasted 1-2 minutes each. We apologize for the issues this incident might have caused to you.
Your product runs on more than Hosted Mender
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services