Hatchet incident

[Shared Cloud] Interruption of Service

Resolved after 21 hours 27 minutes, affecting API (cloud.hatchet.run) and API (Hatchet API), on November 13, 2025.

resolved Nov 14, 2025, 08:06 PM UTC

On November 13, 2025, beginning at approximately 4:55 PM EST, our shared cloud environment experienced disruptions to queueing and monitoring for approximately one hour. The root cause was identified as a RabbitMQ memory saturation issue, which caused the message broker to enter an unrecoverable state. We have since restored full service, implemented monitoring improvements, and are working on several action items to prevent recurrence. This primarily impacted Free, Starter and Growth tier tenants, and did not impact tenants on the enterprise or dedicated plans.

Started
Nov 13, 2025, 10:39 PM UTC
Resolved
Nov 14, 2025, 08:06 PM UTC
Duration
21h 27m
Detected by Pingoru
Nov 13, 2025, 10:39 PM UTC

Affected components

API (cloud.hatchet.run)API (Hatchet API)

Update timeline

  1. investigating Nov 13, 2025, 10:39 PM UTC

    We have received reports for issues on enqueuing and assigning runs, we are currently investigating and working on a resolution.

  2. investigating Nov 13, 2025, 10:55 PM UTC

    We have identified the issue and deployed a patch, services should be fully restored. We are continuing to monitor and will provide a detailed update shortly.

  3. resolved Nov 14, 2025, 08:06 PM UTC

    On November 13, 2025, beginning at approximately 4:55 PM EST, our shared cloud environment experienced disruptions to queueing and monitoring for approximately one hour. The root cause was identified as a RabbitMQ memory saturation issue, which caused the message broker to enter an unrecoverable state. We have since restored full service, implemented monitoring improvements, and are working on several action items to prevent recurrence. This primarily impacted Free, Starter and Growth tier tenants, and did not impact tenants on the enterprise or dedicated plans.

Your product runs on more than Hatchet

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →