Is Cargo down?

The API is intermittently returning 503s. A request can fail and then succeed on retry. Runs that get through still complete. We're investigating.

A stalled API process was being restarted, and those restarts are what returned the 503. We've stopped restarting a stalled process. A request that hits one can still fail, and then succeed on retry. Runs that get through still complete. We're still looking for what causes the stall.

The 503s have stopped. A request that failed earlier can be retried. Runs that get through continue to complete.

Resolved. DNS lookups from production were slow between 18:10 and 18:32 UTC, and requests that needed those lookups timed out. Workflow runs and batches were delayed, and some failed to save. Data syncs to connected systems failed during the same window. Most of those writes retried successfully once DNS recovered. A smaller set that used up its retries may still be missing; we're checking those. No action needed on your side unless a run or sync from that window looks incomplete. Sorry for the interruption.

The API is returning intermittent errors and timeouts as instances restart. Requests may fail and then succeed on retry. We're investigating.

Resolved. Errors stopped at 15:30 UTC. A large export could consume enough memory to take an instance down with it, which is what drove the restarts. Impact was intermittent rather than total: requests that landed on a healthy instance went through, and no data was lost. Instances settled fully over the evening. We're changing exports to stream to storage rather than build the whole file in memory, which removes this failure mode. Sorry for the errors you hit.