Buildkite Outage History

Started Jan 06, 2026, 01:07 AM UTC· resolved Jan 06, 2026, 02:39 AM UTC

1. investigating Jan 06, 2026, 01:07 AM UTC

   We've identified that the clusters index page performing poorly. We're currently investigating the issue, and will provide an update soon.

2. identified Jan 06, 2026, 01:43 AM UTC

   We've identified this issue. We're deploying a fix.

3. monitoring Jan 06, 2026, 02:05 AM UTC

   We've deployed a fix and verified that latency has improved. We are continuing to monitor.

4. resolved Jan 06, 2026, 02:39 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Nov 21, 2025, 04:11 AM UTC· resolved Nov 21, 2025, 05:03 AM UTC

1. identified Nov 21, 2025, 04:11 AM UTC

   We've spotted a slightly elevated error rate when viewing logs in builds UI.

2. monitoring Nov 21, 2025, 04:19 AM UTC

   We've resolved the issue, and error rates when viewing job logs in the UI have returned to normal.

3. resolved Nov 21, 2025, 05:03 AM UTC

   We have confirmed that error rates for log retrieval have returned to nominal levels.

Read the full incident report →

Started Nov 10, 2025, 05:22 AM UTC· resolved Nov 10, 2025, 12:00 PM UTC

1. investigating Nov 10, 2025, 05:22 AM UTC

   We have noticed issues with displaying build annotations, which may cause viewing builds in the web UI. We are still investigating and will provide another update shortly.

2. identified Nov 10, 2025, 05:30 AM UTC

   The REST and GraphQL APIs, as well as the Web UI, are currently affected by increased error rates and slow performance. We have identified the cause of the issue, and are working to address it.

3. identified Nov 10, 2025, 06:36 AM UTC

   We have restored service for the majority of customers and are continuing to work on those that remain. We are working through applying a fix but the exact time to fully restored service is not yet clear.

4. identified Nov 10, 2025, 07:24 AM UTC

   We are actively working on a resolution for remaining customers. We do not have an estimated time for resolution at this time.

5. identified Nov 10, 2025, 08:05 AM UTC

   We have restored service for additional customers and are continuing to work on a resolution for remaining affected customers.

6. identified Nov 10, 2025, 08:46 AM UTC

   We have restored service for additional customers and are continuing to work on a resolution for remaining affected customers. We've identified the root cause of the issue: a database index on our annotations table was replaced, but the replacement index isn't being utilized as expected under load, which is causing performance degradation. Thank you for your understanding and patience as we work to resolve this as quickly as possible.

7. monitoring Nov 10, 2025, 09:02 AM UTC

   The issue has been resolved and service has been restored to all remaining customers. We are continuing to monitor to ensure stability. We apologize for the disruption and thank you for your patience.

8. monitoring Nov 10, 2025, 10:46 AM UTC

   All services are currently stable and we continue to monitor system status. We are working to unblock some remaining jobs that were impacted by the incident. Thank you for your continued patience.

9. resolved Nov 10, 2025, 12:00 PM UTC

   This incident has been resolved. All services are operating normally and remaining jobs have been processed. We apologise for the disruption and thank you for your patience.

10. postmortem Nov 18, 2025, 03:11 AM UTC

    ## Service Impact Between 04:48 and 05:30 UTC on November 10th 2025 the Agent API, that serves interactions with the Buildkite Agent, was severely degraded for the majority of customers. This degradation continued until 08:43 UTC for some remaining customers. The Rest API and GraphQL were degraded from 04:48 until 08:43 UTC. Some agents and jobs were left in an inconsistent state as a result of the degradation. Cases where the system could not self-heal required manual intervention, which was completed by 11:56 UTC. ## Incident Summary At 04:48 UTC, a database migration was applied to remove an index on the table that stores annotations in the Pipelines product. Another index was already in place that was meant to replace the removed one, but there was a small number of high traffic queries that were not hitting the new index as expected. At 04:53 UTC, the impact of this removed index was detected, and observed to be queries timing out on the annotations table. In particular, the agent command to create a new annotation was behind many of these timed-out queries. The result of these queries was substantially increased CPU load on the impacted databases, which degraded performance of all Pipelines features. We immediately began recreating the removed database index, which was completed for roughly half our database shards by 05:30 UTC, and resulted in partial service restoration. For the remaining shards, load from ongoing queries to the annotations prevented the efficient recreation of the index. At this point, we split into teams in order to investigate multiple strategies to reduce that load, including disabling annotation functionality in the product, redirecting new annotations to a separate table, and disabling the annotations table through table-level permissions. In the end, the cleanest and most efficient strategy proved to be temporarily revoking permissions from our application user to read from/write to the annotations table. Using this, in addition to increasing `max_parallel_maintenance_workers` in Postgres, we were able to significantly increase the speed of index creation. Between 07:50 UTC and 08:43 UTC, we used this method to finish recreating the index on the remaining database shards. By this point, latency and error rates across the Agent API, Rest API and GraphQL had returned to regular levels. As a result of the incident, a small number of agents and jobs were left in an inconsistent state. Over the following 4 hours, the agents were freed, and job states cleaned up. ## Changes we're making This incident was primarily caused by a database migration that exposed gaps in our review process. We're implementing the following improvements to prevent similar incidents: We've extended our internal tooling that flags risky migrations during code review to now cover index removals. This tooling now instructs engineers to verify that an index is no longer required before deploying any database migration that would delete the index. Database migrations currently apply to all shards simultaneously. We are implementing phased rollouts for high-risk migrations, which will limit blast radius and enable faster restoration if issues arise. We're evaluating tools such as [HypoPG](https://github.com/HypoPG/hypopg) to improve our analysis when creating or removing indexes. We're also working to surface index usage data from [pganalyze](https://pganalyze.com/) more prominently in our development workflow, making it easier for engineers to make informed decisions about index changes. Finally, we will automate the manual cleanup process used to resolve the inconsistent states that impacted some agents, to ensure the system can self-heal from similar issues in the future.

Read the full incident report →

Started Oct 24, 2025, 03:30 AM UTC· resolved Oct 24, 2025, 03:30 AM UTC

1. resolved Nov 07, 2025, 03:26 AM UTC

   Increase in 403 (Forbidden) errors affecting some Packagecloud repositories. Users may experience access denied errors when attempting to read packages.

2. postmortem Nov 07, 2025, 03:26 AM UTC

   ## Service Impact On October 24th, 2025 from approximately 01:53AM UTC until 04:24AM UTC our package download service experienced an outage affecting customer access. This was confirmed to have impacted Debian and RubyGems. ## Incident Summary At 01:13AM UTC we updated the S3 Bucket Policy for an S3 bucket containing customer packages via Terraform as part of a larger re-architecting project. We received customer reports about Debian package download issues from 01:53AM UTC and started doing isolated, per-customer investigations. At 03:40AM UTC we concluded that the issues were not isolated and declared the incident. More members from our team were called to help investigate the problem, where we found that service for pulling RubyGems had also been impacted. Around 04:00 UTC, the new bucket policy deployed at 01:13AM UTC was identified as a main factor to the access failure and was removed. At 04:19AM UTC our team identified that updating the IAM policy via Terraform did not have the expected result of merging with the pre-existing policy, but rather inadvertently overwrote the pre-existing access policies, causing the loss of access permissions necessary for package downloads. Afterwards it became apparent to our team making these changes that this particular behavior is a [known issue](https://github.com/hashicorp/terraform-provider-aws/issues/6334) with the Terraform AWS provider, which contributed to the difficulty of detection and prevention, thereby increasing the likelihood of the incident occurring for those unaware. After redeploying the pre-existing S3 bucket IAM policy, we confirmed service functionality for Debian package downloads was restored at 04:24 AM UTC. The team subsequently verified that pulling RubyGems and other supported package types were also functioning as expected, and the incident was closed at 04:47 AM UTC. ## Changes we're making We are revising our monitoring approaches to identify and address service disruptions proactively, while also improving our communication processes to ensure timely and accurate customer updates.

Read the full incident report →

Started Oct 20, 2025, 08:38 AM UTC· resolved Oct 20, 2025, 11:52 AM UTC

1. investigating Oct 20, 2025, 08:38 AM UTC

   We're tracking an ongoing incident with an upstream provider, and are investigating any impact on Buildkite's services.

2. monitoring Oct 20, 2025, 09:22 AM UTC

   We are continuing to track an incident with AWS' us-east-1 region, though impact to Buildkite's services appears to be minimal.

3. monitoring Oct 20, 2025, 09:53 AM UTC

   We're seeing signs of recovery from AWS. Build notifications may be delayed due to this incident and the flow on effect to other providers but are now being delivered successfully.

4. resolved Oct 20, 2025, 11:52 AM UTC

   With the recovery of service within AWS, we are treating this incident as resolved. Customers running their agents in AWS may experience lingering issues as service within the region stabilizes.

Read the full incident report →

Started Oct 15, 2025, 03:34 AM UTC· resolved Oct 15, 2025, 03:50 AM UTC

1. investigating Oct 15, 2025, 03:34 AM UTC

   We're observing increased latency and error rates for a subset of our customers. We're currently investigating and will provide status updates as they become available.

2. identified Oct 15, 2025, 03:38 AM UTC

   We've identified a problematic deploy and performing a rollback.

3. monitoring Oct 15, 2025, 03:42 AM UTC

   The rollback is finishing and we will continue to monitor the latency and errors.

4. resolved Oct 15, 2025, 03:50 AM UTC

   The rollback has been completed.

Read the full incident report →