Resolved after 31 minutes, affecting Hosted Mender US, on June 23, 2025.
This morning, the operation team performed a planned Redis Cluster upgrade, starting at 04:40 UTC. Around 04:56 UTC, one of the Redis pod got killed because of Out of Memory issues, causing the Device Auth service to experience connection failure. To resolve this, the operation team increased the memory allocated to the Redis Cluster, starting at 05:05 UTC. The change was fully implemented by 05:14 UTC, and no more error log was seen from the Device Auth service, which was returned to normal operation.
Affected components
Update timeline
- investigating Jun 23, 2025, 05:08 AM UTC
We are investigating an issue regarding Redis cluster and the Device Auth Service which is in degraded state.
- monitoring Jun 23, 2025, 05:19 AM UTC
The issue has been identified: a new Redis pod was restarting because of OOMKill. More memory has been given to the Redis pool and now the services are up. We're monitoring the result.
- resolved Jun 23, 2025, 05:39 AM UTC
This incident has been resolved.
- postmortem Jun 23, 2025, 08:29 AM UTC
This morning, the operation team performed a planned Redis Cluster upgrade, starting at 04:40 UTC. Around 04:56 UTC, one of the Redis pod got killed because of Out of Memory issues, causing the Device Auth service to experience connection failure. To resolve this, the operation team increased the memory allocated to the Redis Cluster, starting at 05:05 UTC. The change was fully implemented by 05:14 UTC, and no more error log was seen from the Device Auth service, which was returned to normal operation.
Your product runs on more than Hosted Mender
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services