Fly.io incident

Setting secrets and creating apps is degraded

Resolved after 33 minutes, affecting Machines API and Deployments, on March 11, 2026.

monitoring Mar 11, 2026, 11:03 AM UTC

While the secret storage service was in a read-only state, app creation requests queued up, due to the retry logic and insufficient request concurrency limits in our GraphQL API. This prevented our GraphQL API from serving any other requests. We have scaled up the GraphQL API and are continuing to monitor the situation.

Started: Mar 11, 2026, 11:03 AM UTC
Resolved: Mar 11, 2026, 11:37 AM UTC
Duration: 33m
Detected by Pingoru: Mar 11, 2026, 11:03 AM UTC

Affected components

Machines APIDeployments

Update timeline

identified Mar 11, 2026, 09:19 AM UTC

An ongoing data migration in our secret storage service is causing degraded Machines API functionality.
monitoring Mar 11, 2026, 10:14 AM UTC

A fix has been implemented and we are monitoring the results.
monitoring Mar 11, 2026, 11:03 AM UTC

While the secret storage service was in a read-only state, app creation requests queued up, due to the retry logic and insufficient request concurrency limits in our GraphQL API. This prevented our GraphQL API from serving any other requests. We have scaled up the GraphQL API and are continuing to monitor the situation.
resolved Mar 11, 2026, 11:37 AM UTC

This incident has been resolved.

Your product runs on more than Fly.io

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.