Hatchet incident

[us-west-2] Increased Latency on Dashboard and APIs

Resolved after 1 hour 7 minutes, affecting API (cloud.hatchet.run) and API (Hatchet API) and 1 more, on July 1, 2026.

resolved Jul 01, 2026, 03:50 PM UTC

We're continuing to monitor the fix. We've been seeing significant recovery since about 35 minutes ago (15:15 UTC). If you're seeing tasks that are stuck during execution, cancelling and replaying will likely get them moving again.

Started
Jul 01, 2026, 02:43 PM UTC
Resolved
Jul 01, 2026, 03:50 PM UTC
Duration
1h 7m
Detected by Pingoru
Jul 01, 2026, 02:43 PM UTC

Affected components

API (cloud.hatchet.run)API (Hatchet API)Website (hatchet.run)

Update timeline

  1. investigating Jul 01, 2026, 02:43 PM UTC

    We're monitoring some degradation in performance on the Hatchet dashboard and REST APIs, and the scheduler, and are working on a fix. We believe about 30% of tenants are affected

  2. investigating Jul 01, 2026, 03:00 PM UTC

    We're continuing to monitor affected gRPC APIs as well. We're seeing some reports of StatusCode.UNAVAILABLE being returned to SDKs from the API

  3. investigating Jul 01, 2026, 03:15 PM UTC

    We've applied a candidate fix and are seeing some services begin to recover, and are continuing to monitor

  4. resolved Jul 01, 2026, 03:50 PM UTC

    We're continuing to monitor the fix. We've been seeing significant recovery since about 35 minutes ago (15:15 UTC). If you're seeing tasks that are stuck during execution, cancelling and replaying will likely get them moving again.

Your product runs on more than Hatchet

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →