Hatchet Outage History

Started Aug 22, 2026, 04:24 PM UTC· resolved Aug 22, 2026, 09:27 PM UTC

1. investigating Aug 22, 2026, 04:24 PM UTC

   On August 22, 2026, starting around 12:00 UTC, the shared us-west-2 cloud environment experienced elevated database CPU and elevated scheduling latency. A high volume of concurrent workflow runs with stacked concurrency controls on a shared shard caused lock contention and delayed task processing for tenants on that environment. We took action to throttle the work driving saturation and constraining scheduler capacity while the system recovered. Database load and scheduling latency returned to normal later the same afternoon. No data loss occurred.

2. resolved Aug 22, 2026, 09:27 PM UTC

   On August 22, 2026, starting around 12:00 UTC, one shared shard in the us-west-2 cloud environment experienced elevated database CPU and elevated scheduling latency. A high volume of concurrent workflow runs with stacked concurrency controls caused lock contention and delayed task processing for tenants on that shard. Other shared shards in us-west-2, shared environments in other regions, and dedicated customer shards were not affected. Took steps to throttle the workload and reduce contention. Database load and scheduling latency on the affected shard returned to normal later the same afternoon. No data loss occurred.

3. investigating Aug 22, 2026, 09:27 PM UTC

   On August 22, 2026, starting around 12:00 UTC, one shared shard in the us-west-2 cloud environment experienced elevated database CPU and elevated scheduling latency. A high volume of concurrent workflow runs with stacked concurrency controls caused lock contention and delayed task processing for tenants on that shard. **Other shared shards in us-west-2, shared environments in other regions, and dedicated customer shards were not affected.** Took steps to throttle the workload and reduce contention. Database load and scheduling latency on the affected shard returned to normal later the same afternoon. No data loss occurred.

Read the full incident report →

Started Jul 29, 2026, 07:56 AM UTC· resolved Jul 29, 2026, 08:27 AM UTC

1. investigating Jul 29, 2026, 07:56 AM UTC

   We're monitoring increased latency on a single shared cloud shard in our us-west-2 region experienced elevated database lock contention, which is causing higher API and scheduling latency and some intermittent task-dispatch delays for tenants on that shard. Tasks are continuing to be ingested and assigned.

2. investigating Jul 29, 2026, 08:20 AM UTC

   We have pushed some configuration changes and are seeing initial signs of improvement. We will continue to monitor the situation.

3. resolved Jul 29, 2026, 08:27 AM UTC

   We saw elevated database load on our shared production environment beginning around 05:00 UTC, which caused some multi-step workflows to experience scheduling delays and increased API/dashboard latency for affected tenants. We reduced connection pressure on the shared database and latency has returned to normal. We’re continuing to monitor closely.

Read the full incident report →

Started Jul 24, 2026, 11:14 AM UTC· resolved Jul 24, 2026, 11:14 AM UTC

1. resolved Jul 24, 2026, 11:14 AM UTC

   Between approximately 05:00–10:00 UTC on July 24, the shared cloud shard in our us-west-2 region experienced elevated database lock contention, which caused higher API and scheduling latency and some intermittent task-dispatch delays for tenants on that shard. This was isolated to a single shard in us-west-2 (cloud). Other regions and shards were not affected. We have no evidence of dropped workload, and all assignments were eventually completed. We are continuing to investigate and monitor.

Read the full incident report →

Started Jul 01, 2026, 02:43 PM UTC· resolved Jul 01, 2026, 03:50 PM UTC

1. investigating Jul 01, 2026, 02:43 PM UTC

   We're monitoring some degradation in performance on the Hatchet dashboard and REST APIs, and the scheduler, and are working on a fix. We believe about 30% of tenants are affected

2. investigating Jul 01, 2026, 03:00 PM UTC

   We're continuing to monitor affected gRPC APIs as well. We're seeing some reports of StatusCode.UNAVAILABLE being returned to SDKs from the API

3. investigating Jul 01, 2026, 03:15 PM UTC

   We've applied a candidate fix and are seeing some services begin to recover, and are continuing to monitor

4. resolved Jul 01, 2026, 03:50 PM UTC

   We're continuing to monitor the fix. We've been seeing significant recovery since about 35 minutes ago (15:15 UTC). If you're seeing tasks that are stuck during execution, cancelling and replaying will likely get them moving again.

Read the full incident report →

Started Jul 01, 2026, 04:14 AM UTC· resolved Jul 01, 2026, 05:42 AM UTC

1. investigating Jul 01, 2026, 04:14 AM UTC

   We are investigating reports of elevated latency on our OLAP database, which powers parts of the Hatchet UI and some API requests. The core database is not affected. Workflow execution and workloads are continuing to run normally.

2. resolved Jul 01, 2026, 05:42 AM UTC

   We have identified the source of the latency and have pushed initial configuration changes to mitigate. We will continue to monitor for degraded performance.

Read the full incident report →

Started Jun 19, 2026, 11:00 PM UTC· resolved Jun 21, 2026, 08:00 PM UTC

1. investigating Jun 19, 2026, 11:00 PM UTC

   Hatchet's shared cloud (free, team, and scale plans) in us-west-2 experienced a partial outage between 6/19/26 23:07:00 UTC and 6/20 and 00:34:00 UTC caused by crashes on an internal queue. We were alerted and responded immediately, which brought the service online. Task processing (e.g. dispatching tasks to workers) was affected, but ingestion was not. You may see stuck DAGs, which likely will need to be cancelled and replayed or retriggered from scratch to reconcile. We'll continue to update as we investiage the root cause of the message queue crash.

2. resolved Jun 21, 2026, 08:00 PM UTC

   We’re back! Hatchet Cloud region us-west-2 should be up and running. Thanks for bearing with us.

Read the full incident report →

Started May 13, 2026, 06:20 PM UTC· resolved May 13, 2026, 06:20 PM UTC

1. resolved May 13, 2026, 06:20 PM UTC

   Hatchet experienced ~15 minutes of partial, unexpected downtime on some parts of the system on May 13th, 2026 between 17:56 and 18:20 UTC, causing delayed task runs, the dashboard to appear frozen, and errors on ingestion in certain cases. Cron runs were also impacted, and will need to be replayed manually (which we can help with, if needed). This was an isolated incident, which should not happen again. Root Cause: At approximately 17:56, we began a routine deploy included a database migration. This migration ran an ALTER TABLE statement before a VALIDATE CONSTRAINT in the same transaction, which resulted in the ACCESS EXCLUSIVE lock on the table that was acquired during the original ALTER TABLE to be held for the duration of the transaction. This lock was then held for the duration. of the VALIDATE CONSTRAINT (which would otherwise not require an ACCESS EXCLUSIVE lock). This kept the internal table locked for an extended period of time, causing significant backpressure on our internal queue, and some upstream effects on publishers. We took steps to mitigate the effects of the issue as soon as we were aware of it, at which point full recovery took approximately ten minutes. At this time, we believe that task ingestion was continuing to progress as normal, albeit with delays on execution, as long as the task was accepted by Hatchet. Action Items: We’re actively working on an improved way of running migrations under load on large databases, and are planning to have that ready to use within the upcoming weeks, to help safeguard against this kind of unexpected failure scenario in our deploy process. Additionally, we’re continuously working on improving our monitoring and alerting to help recover more quickly from incidents such as this one.

Read the full incident report →

Started Mar 25, 2026, 04:24 PM UTC· resolved Mar 25, 2026, 04:24 PM UTC

1. resolved Mar 25, 2026, 04:24 PM UTC

   Between 5:00 UTC and 16:25 UTC, we received reports of increased latency and error rates impacting some tenants in us-west-2. We have scaled capacity and all tenants are recovered.

Read the full incident report →

Started Dec 04, 2025, 02:43 PM UTC· resolved Dec 04, 2025, 03:04 PM UTC

1. investigating Dec 04, 2025, 02:43 PM UTC

   We are seeing DNS issues with our upstream database provider which is preventing runs from loading in the UI or over the API. Core queueing operations are not impacted, but endpoints utilizing the REST API will fail to respond.

2. investigating Dec 04, 2025, 02:57 PM UTC

   Update: we are rolling out a hotfix to get around the DNS resolution failures. Services are starting to recover.

3. resolved Dec 04, 2025, 03:04 PM UTC

   This issue is now resolved and runs should be available again over the UI.

Read the full incident report →

Started Nov 18, 2025, 01:33 PM UTC· resolved Nov 18, 2025, 01:55 PM UTC

1. investigating Nov 18, 2025, 01:33 PM UTC

   We are currently tracking a global outage of Cloudflare's DNS. This is currently impacting access to cloud.onhatchet.run and dashboards for dedicated tenants as well as REST API calls. Workers, which connect via gRPC and do not proxy through Cloudflare, should continue to run and should not be impacted. You can track this issue directly here: https://www.cloudflarestatus.com/

2. resolved Nov 18, 2025, 01:55 PM UTC

   We have taken steps to mitigate affected DNS services and have fully restored all Hatchet services

Read the full incident report →

Started Nov 13, 2025, 10:39 PM UTC· resolved Nov 14, 2025, 08:06 PM UTC

1. investigating Nov 13, 2025, 10:39 PM UTC

   We have received reports for issues on enqueuing and assigning runs, we are currently investigating and working on a resolution.

2. investigating Nov 13, 2025, 10:55 PM UTC

   We have identified the issue and deployed a patch, services should be fully restored. We are continuing to monitor and will provide a detailed update shortly.

3. resolved Nov 14, 2025, 08:06 PM UTC

   On November 13, 2025, beginning at approximately 4:55 PM EST, our shared cloud environment experienced disruptions to queueing and monitoring for approximately one hour. The root cause was identified as a RabbitMQ memory saturation issue, which caused the message broker to enter an unrecoverable state. We have since restored full service, implemented monitoring improvements, and are working on several action items to prevent recurrence. This primarily impacted Free, Starter and Growth tier tenants, and did not impact tenants on the enterprise or dedicated plans.

Read the full incident report →

Started Aug 09, 2025, 01:55 AM UTC· resolved Aug 09, 2025, 01:55 AM UTC

1. resolved Aug 09, 2025, 01:55 AM UTC

   We saw an increased number of 503 error rates between 1:40-1:50 UTC. We've deployed a fix and are monitoring now.

Read the full incident report →

Started Dec 11, 2024, 07:11 PM UTC· resolved Dec 11, 2024, 07:11 PM UTC

1. resolved Dec 11, 2024, 07:11 PM UTC

   The Hatchet API is currently experiencing increased API error rates.

Read the full incident report →