Is Buildkite down?

We are investigating elevated job dispatch rates that is impacting a subset of customers.

We've identified the cause and are currently applying mitigations

We've spotted that something has gone wrong. We've applied the mitigation and currently monitoring the issue, and will provide an update soon.

We’ve applied mitigations and are seeing recovery. We’re continuing to monitor the issue

We are seeing the job dispatch times are back to normal.

We saw increased latency and error rates in REST and GraphQL for a subset of our customers starting around 22:30 UTC until 22:45. The platform has been stable since. We're investigating the root cause

We identified that the majority of the impact was for a single customer and was caused by a database failover. We are investigating the cause and we will work with the impacted customer.

Between 17:07-17:19 UTC, our API endpoints experienced elevated latencies and errors due to issues with a shared service. Our monitoring systems detected these issues, which triggered automated system remediations and resulted in the service self-healing. Although our APIs are once again stable, we are continuing to investigate the cause of this very brief service degradation.

Between 23:16 and 23:27 we saw elevated error rates across our platform due to a scheduled database upgrade. That upgrade was not intended to cause any impact, but due to an unforeseen dependency on this database customers experienced error rates averages 2.5% during this period.

We've received reports that Hosted Agents jobs are delayed in starting and are investigating.

We've identified the issue and have applied remediations. We are already beginning to see recovery in Hosted Agents job processing times for affected customers.

We continue to see improvements and are monitoring processing of the Hosted Agents jobs backlog.

Processing of Hosted Agents jobs are no longer delayed.