Upstash incident

Fly.io Upstash Redis Service Distruption

Resolved after 3 hours 59 minutes, on May 12, 2026.

postmortem May 15, 2026, 12:43 PM UTC

On May 12th and 13th at various times, a subset of Upstash Redis instances on [Fly.io](http://Fly.io) experienced intermittent hangs and elevated error rates. The Redis process would stall inside a logging syscall — alive but not making progress — which made the issue hard to spot from our usual telemetry. After investigating with Fly's team, we identified the root cause as a bad interaction between a recent guest kernel update on Fly's newer machines and an upstream Cloud Hypervisor bug \([cloud-hypervisor#7672](https://github.com/cloud-hypervisor/cloud-hypervisor/issues/7672)\) affecting log writes from inside the VM. We mitigated by disabling the affected logging paths, and Fly has since rolled out a hypervisor-side patch, fully resolving the issue. No data was lost. Sorry for the disruption.

Started
May 12, 2026, 09:49 AM UTC
Resolved
May 12, 2026, 01:48 PM UTC
Duration
3h 59m
Detected by Pingoru
May 12, 2026, 09:49 AM UTC

Update timeline

  1. investigating May 12, 2026, 09:49 AM UTC

    Some regions are experiencing connectivity issues due to an ongoing network problem. We are currently investigating

  2. identified May 12, 2026, 10:50 AM UTC

    The issue has been identified and the fix is being implemented.

  3. monitoring May 12, 2026, 12:05 PM UTC

    A fix has been implemented and we are monitoring the results.

  4. resolved May 12, 2026, 01:48 PM UTC

    This incident has been resolved.

  5. postmortem May 15, 2026, 12:43 PM UTC

    On May 12th and 13th at various times, a subset of Upstash Redis instances on [Fly.io](http://Fly.io) experienced intermittent hangs and elevated error rates. The Redis process would stall inside a logging syscall — alive but not making progress — which made the issue hard to spot from our usual telemetry. After investigating with Fly's team, we identified the root cause as a bad interaction between a recent guest kernel update on Fly's newer machines and an upstream Cloud Hypervisor bug \([cloud-hypervisor#7672](https://github.com/cloud-hypervisor/cloud-hypervisor/issues/7672)\) affecting log writes from inside the VM. We mitigated by disabling the affected logging paths, and Fly has since rolled out a hypervisor-side patch, fully resolving the issue. No data was lost. Sorry for the disruption.

Your product runs on more than Upstash

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →