Hatchet incident

[us-west-2] Increased latency on single shard

Resolved after 31 minutes, affecting API (cloud.hatchet.run) and API (Hatchet API), on July 29, 2026.

resolved Jul 29, 2026, 08:27 AM UTC

We saw elevated database load on our shared production environment beginning around 05:00 UTC, which caused some multi-step workflows to experience scheduling delays and increased API/dashboard latency for affected tenants. We reduced connection pressure on the shared database and latency has returned to normal. We’re continuing to monitor closely.

Started
Jul 29, 2026, 07:56 AM UTC
Resolved
Jul 29, 2026, 08:27 AM UTC
Duration
31m
Detected by Pingoru
Jul 29, 2026, 07:56 AM UTC

Affected components

API (cloud.hatchet.run)API (Hatchet API)

Update timeline

  1. investigating Jul 29, 2026, 07:56 AM UTC

    We're monitoring increased latency on a single shared cloud shard in our us-west-2 region experienced elevated database lock contention, which is causing higher API and scheduling latency and some intermittent task-dispatch delays for tenants on that shard. Tasks are continuing to be ingested and assigned.

  2. investigating Jul 29, 2026, 08:20 AM UTC

    We have pushed some configuration changes and are seeing initial signs of improvement. We will continue to monitor the situation.

  3. resolved Jul 29, 2026, 08:27 AM UTC

    We saw elevated database load on our shared production environment beginning around 05:00 UTC, which caused some multi-step workflows to experience scheduling delays and increased API/dashboard latency for affected tenants. We reduced connection pressure on the shared database and latency has returned to normal. We’re continuing to monitor closely.

Your product runs on more than Hatchet

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →