Buildkite Outage History

Started Oct 22, 2026, 04:51 PM UTC· resolved Oct 22, 2026, 05:37 PM UTC

1. investigating Oct 22, 2026, 04:51 PM UTC

   We've received reports that Hosted Agents jobs are delayed in starting and are investigating.

2. identified Oct 22, 2026, 05:09 PM UTC

   We've identified the issue and have applied remediations. We are already beginning to see recovery in Hosted Agents job processing times for affected customers.

3. monitoring Oct 22, 2026, 05:22 PM UTC

   We continue to see improvements and are monitoring processing of the Hosted Agents jobs backlog.

4. resolved Oct 22, 2026, 05:37 PM UTC

   Processing of Hosted Agents jobs are no longer delayed.

Read the full incident report →

Started Oct 20, 2026, 10:13 PM UTC· resolved Oct 21, 2026, 10:38 PM UTC

1. investigating Oct 20, 2026, 10:13 PM UTC

   The private keys for eight certificates issued by eMudhra, an external certificate authority, have been compromised. Software that still trusts these certificates may accept fraudulent HTTPS certificates. We’re updating the services that power Buildkite and preparing new CLI and Agent releases, which we expect to ship shortly. We’re also tracking fixes to the libraries and trust stores we depend on and will apply them as they become available. We’ll update this page with the new versions and any action customers need to take. Mozilla has published background on the compromise.

2. monitoring Oct 20, 2026, 11:25 PM UTC

   We're continuing to monitor the situation, and are making releases to many of our components that run in customer infrastructure. At time of writing, we've released versions that distrust the affected eMudhra certificates for: - [Buildkite agent](https://github.com/buildkite/agent): - [v4.7.0](https://github.com/buildkite/agent/releases/tag/v4.7.0) - [v3.138.2](https://github.com/buildkite/agent/releases/tag/v3.138.2) - [Buildkite CLI](https://github.com/buildkite/cli/releases/tag/v3.64.2) - [bktec](https://github.com/buildkite/test-engine-client/releases/tag/v3.3.1) We recommend updating to these versions to prevent exposure to the affected certificate providers. We will continue to issue updates for the other binaries we distribute, and are updating any internal services that are affected.

3. monitoring Oct 21, 2026, 05:07 AM UTC

   We’ve finished updating the services that power Buildkite so they no longer trust the affected eMudhra certificates. No action is needed from customers for Buildkite hosted services. For components that run in customer infrastructure, we still recommend upgrading to the releases listed in our previous update (Buildkite agent v4.7.0 / v3.138.2, Buildkite CLI v3.64.2, and bktec v3.3.1). We’re continuing to release updates for the other binaries we distribute and will post them here as they ship.

4. resolved Oct 21, 2026, 10:38 PM UTC

   All services that power Buildkite have been updated to distrust the affected certificates, and updated releases are available for the components we distribute that run in customer infrastructure: Buildkite agent: v4.7.0 and v3.138.2 Buildkite CLI: v3.64.2 bktec: v3.3.1 elastic-ci-stack-for-aws: v7.8.0 agent-stack-k8s: v0.52.0 We recommend updating to these versions to prevent exposure to the affected certificate providers.

Read the full incident report →

Started Oct 20, 2026, 07:57 PM UTC· resolved Oct 20, 2026, 09:05 PM UTC

1. investigating Oct 20, 2026, 07:57 PM UTC

   We are investigating delays to build and job notifications for a subset of customers.

2. monitoring Oct 20, 2026, 08:15 PM UTC

   We've identified the issue, and added some additional capacity. Notification latency has returned to normal levels. We are monitoring for stability.

3. resolved Oct 20, 2026, 09:05 PM UTC

   Adding additional capacity has resolved the notifications latency.

Read the full incident report →

Started Oct 19, 2026, 03:45 PM UTC· resolved Oct 19, 2026, 05:12 PM UTC

1. resolved Oct 19, 2026, 05:39 PM UTC

   We are getting reports from a few customers that Hosted Agent jobs are delayed in starting. We are investigating.

Read the full incident report →

Started Oct 06, 2026, 06:19 PM UTC· resolved Oct 06, 2026, 11:59 PM UTC

1. investigating Oct 06, 2026, 06:19 PM UTC

   We've received reports of degraded checkout performance with GitHub repositories when using Hosted Agents and are investigating.

2. identified Oct 06, 2026, 07:07 PM UTC

   We continue to observe degraded checkout performance. Our engineering teams are working on establishing an alternative network transit path to alleviate networking performance issues.

3. resolved Oct 06, 2026, 11:59 PM UTC

   The issue is now resolved. Network maintenance from our provider caused unexpected side effects that impacted outgoing traffic from Hosted Agents.

Read the full incident report →

Started Oct 01, 2026, 06:10 AM UTC· resolved Oct 01, 2026, 06:57 AM UTC

1. identified Oct 01, 2026, 06:10 AM UTC

   Since 2026-09-30 20:52:36 UTC on 30 September, small amount of incoming GitHub webhook deliveries to Buildkite pipelines are being rejected, so the related builds are not created. For people who use GitHub.com may affect ability to receive status checks. We have identified the cause and are deploying a mitigation. This does not affect GitHub Enterprise Server and other SCM providers.

2. monitoring Oct 01, 2026, 06:33 AM UTC

   The mitigation is rolling out and we are seeing recovery. We will continue to monitor and provide an update.

3. resolved Oct 01, 2026, 06:57 AM UTC

   The error rate has returned to normal levels.

Read the full incident report →

Started Sep 28, 2026, 01:30 AM UTC· resolved Sep 28, 2026, 01:30 AM UTC

1. resolved Sep 30, 2026, 06:30 AM UTC

   Between 11:45 and 15:00 (GMT+10) on 28 September, macOS hosted agent jobs could not start because of a misconfigured agent image update. This has been fixed and new jobs are running normally. Jobs from that window still waiting in the Scheduled state won't start on their own. Please cancel and retry them.

Read the full incident report →

Started Sep 22, 2026, 08:07 PM UTC· resolved Sep 22, 2026, 08:50 PM UTC

1. investigating Sep 22, 2026, 08:07 PM UTC

   We're seeing an issue affecting webhook ingestion and execution of GraphQL queries and are investigating the cause.

2. identified Sep 22, 2026, 08:26 PM UTC

   Our engineers have addressed the issue and we're monitoring recovery.

3. resolved Sep 22, 2026, 08:50 PM UTC

   Services are fully recovered. Webhook ingestion was affected between 19:09 UTC - 19:32 UTC.

4. postmortem Sep 28, 2026, 04:12 AM UTC

   ## Service Impact On September 22, 2026, between 19:09 and 19:32 UTC, customers experienced failures ingesting webhooks and executing GraphQL queries. Some web application requests also failed or timed out. During this period, 2.0% of requests to our webhook ingestion endpoints and 20.3% of requests to our external GraphQL endpoint returned HTTP 500 errors. Webhook ingestion errors ceased by 19:30 UTC, and external GraphQL errors returned to background levels by 19:32 UTC. ## Incident Summary While investigating a separate API performance issue, we made a manual configuration change intended to rollback a routing change for one database proxy. The command instead changed the routing configuration for another database proxy, disrupting access to a database shard used internally by Buildkite. Some customer-facing operations search across all database shards to locate their shard, where routing information isn't available in the request. Those requests round robin through available shards, and could encounter the unavailable shard and wait for a database connection to time out. This extended the impact beyond our internal workloads. We reverted the incorrect configuration and verified that the database proxy had healthy endpoints and load-balancer targets. Two contributing factors allowed the change to cause an outage: the command was not independently checked before execution, and our infrastructure accepted a Service configuration that selected the wrong database proxy workload. The cross-shard lookup behaviour then allowed the disruption to affect requests for other customers. This configuration-change outage was separate from the API performance issue we were originally investigating. ## Changes we're making * Safer operational procedures: We have updated our runbook and introduced an explicit second person review requirement for manual production commands. * Automated safeguards: We’ve added validation to prevent database proxy routing changes from selecting the wrong workload. * Improved failure isolation: We’ve been working on an ongoing initiative to make database shard discovery more resilient, eliminating the remaining cases where an unavailable shard may impact requests for other customers.

Read the full incident report →

Started Sep 21, 2026, 11:47 PM UTC· resolved Sep 22, 2026, 12:32 AM UTC

1. monitoring Sep 21, 2026, 11:47 PM UTC

   We're aware of a major AWS outage in us-east-1, we are taking action to prevent impact and monitoring the situation.

2. resolved Sep 22, 2026, 12:32 AM UTC

   We did not experience any customer impact during this AWS us-east-1 outage.

Read the full incident report →

Started Sep 21, 2026, 10:35 PM UTC· resolved Sep 21, 2026, 11:46 PM UTC

1. investigating Sep 21, 2026, 10:35 PM UTC

   We're seeing increased latency and error rates for a subset of our customers. We're currently investigating and will provide status updates as they become available.

2. monitoring Sep 21, 2026, 11:12 PM UTC

   We observed a slight increase in latency for build creation isolated to a few customers, this has now passed and we are seeing signs of improvement.

3. resolved Sep 21, 2026, 11:46 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 09, 2026, 07:07 PM UTC· resolved Sep 10, 2026, 01:54 AM UTC

1. investigating Sep 09, 2026, 07:07 PM UTC

   We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.

2. investigating Sep 09, 2026, 07:41 PM UTC

   We are continuing to investigate this issue. We are seeing impact on the Agent API which will affect job scheduling, artifact uploads, and an increase in 5xx responses from the Agent API endpoints.

3. investigating Sep 09, 2026, 08:28 PM UTC

   We are continuing to investigate elevated error rates across multiple services, and are working to determine the cause.

4. monitoring Sep 09, 2026, 08:53 PM UTC

   We are seeing improvements across the affected services, and are seeing services return to normal functionality. We are continuing to monitor and are determining the root cause.

5. resolved Sep 10, 2026, 01:54 AM UTC

   We have seen full recovery for customers since 20:28 UTC. We experienced an autoscaling feedback loop which increased the number of connections to our redis cluster above its ability to respond. This had widespread impact for all of our customers with Web UI, Agent API, REST API and job queue impact between 18:43-19:21 UTC, and again between 20:02-20:28 UTC. A full post incident review will be available later this week.

6. postmortem Sep 11, 2026, 06:03 AM UTC

   ## Service Impact _All times UTC unless stated otherwise._ Customers experienced elevated errors and latency across the Buildkite web interface, REST API, Agent API, and job queue during two periods: 18:43:00–19:21:00 and 20:02:00–20:28:00 on September 9, 2026. During these periods, customers encountered failed API requests, delayed job dispatch, and errors in agent operations including authentication, job acceptance, and artifact uploads. Some jobs impacted by the incident did not recover automatically. These jobs had to be manually cancelled and retried to complete successfully. Between 18:43:00–19:21:00: Agent API had an error rate up to 11.2% with no latency impact. REST API had a 9.7% error rate with no latency impact. Web request had a 2.6% error rate with no latency impact. Job queues were delayed by up to 3 minutes. Between 20:02:00–20:28:00: Agent API had an error rate up to 24.7% with no latency impact. REST API had a 4.9% error rate with no latency impact. Web requests had a 0.8% error rate with no latency impact. Job queues were delayed by up to 10 minutes. During this window, notifications were also delayed up to 1 minute. Core services fully recovered by 20:28:00. ## Incident Summary ### Background Several Buildkite services use a shared Redis cluster for coordination, caching, and agent-facing operations. This is one of the last pieces of non-shard aligned shared infrastructure, and is on our roadmap to address. AWS enforces a connection limit of 65,000 connections per node on this Redis cluster. As part of our ongoing migration from Amazon ECS to Amazon EKS, we reduced the number of Ruby threads in each Puma and Sidekiq process to their framework defaults. In ECS, we historically configured high thread counts, which caused resource contention and increased latency without a corresponding increase in throughput. We’d intentionally timed this work as part of the EKS migration because the migration already required us to redesign how these services are sized and scaled: EKS can respond directly to request queues and worker utilization, allowing us to run more, smaller application containers while preserving total capacity. This gives customers faster, more predictable API responses and background-job processing, particularly during periods of high load. ### What happened We began testing Puma Agent services on EKS in July and migrated traffic incrementally from late July, completing the migration earlier this week. Running more, smaller processes increased total Redis connection demand—a trade-off we had modeled, but not adequately accounted for before completing the migration. Before the incident, our shared services were already using much of the available connection capacity on the impacted Redis cluster, around 60-80% throughout a 24 hour window. At 18:43:10 we started seeing elevated HTTP request errors. At this time, our application reported a burst of Redis connection timeouts, all targeting a single replica in the cluster. Clients with active Redis connections then attempted to reconnect. Failed connections were retried, causing a surge in connection attempts that exhausted that same Redis host’s connection-tracking allowance. This caused the host to reject new connections from the Agent API’s application pods, which led to increased request latency for our Puma Agent service. At 18:43:30, the increased request latency from this network failure caused our application platform to rapidly start more Puma Agent containers to respond to the increase in request latency. Within a minute, we’d doubled our running pod count. Each additional container opened its own pool of Redis connections. At 18:44:00, several Redis nodes hit their limit of 65,000 connections, and latency of requests slowed further triggering more Puma Agent autoscaling. Once the Redis connection limits were reached, applications could no longer reliably connect to Redis. This caused failures across agent authentication, job assignment, artifact uploads, and other API operations. The failures then created a feedback loop: slower requests caused our shard aligned Puma Agent services to autoscale, and those additional containers opened more Redis connections, placing further pressure on Redis. By 18:50:00, our system stabilized at max, running about five times the number of pods we’d started with. This stable state allowed the autoscaling to start bringing pod counts back down. By 19:10:00 the error rate had returned to 0 and and services were no longer degraded. By 19:20:00 our pod counts had autoscaled down to more reasonable but still high numbers, and the Redis cluster was functioning again. At this point customers saw recovery. At 20:02:00, a routine application deployment temporarily introduced additional containers. Since the Redis cluster was already under strain, this quickly increased request latency which added autoscaling on top of the regular deployment surge numbers. This amplified the connection growth again and caused the second period of degraded service, lasting until 20:28:00. The influx of new pods resulted in a very large number of Envoy Gateway configuration updates, which in turn caused some network throttling of the underlying node. This caused packet drops and delays updating our Cilium Operator, which controls configuration for the cluster’s overlay network. This caused resulted some Buildkite Jobs to remain in a stuck state for some customers, requiring manual retries. ### How we responded When we saw that the Redis cluster had reached its capacity to handle incoming connections, we immediately began adding more nodes to the cluster, increasing incoming connection capacity by 33%. When we saw the impact of deployments, we immediately paused further application deployments while we investigated the connection growth. We increased the Puma Agent utilization autoscaling threshold, reducing the rate at which workloads scale in response to brief load increases. We also increased our scale up stabilization window to prevent future runaway autoscaling of our workloads. We reduced the configured maximum surge of our workloads during application deployments, to reduce pod churn impact on connections. Since Redis connections and latency remained stable, we then resumed deployments while continuing to monitor the cluster. ## Changes we're making **Reduce autoscaling sensitivity.** * We increased the utilization threshold that triggers the Puma Agent autoscaling, so short-lived latency increases do not cause rapid increases in application and Redis connection demand. * We increased our scale up stabilization window so that we only add more pods in response to scaling triggers in incremental bursts. This will throttle scaling up based on downstream latency and runaway scaling. **Reduce deployment-related load.** * We reduced the maximum deployment surge for Puma and Sidekiq workloads, limiting the additional capacity and downstream connections introduced during deployments. **Increase immediate Redis capacity by 33%.** * We added more nodes to the Redis cluster to handle more connections, given the shape of our connection load has shifted with our different thread counts in EKS **Reduce connection pressure from high-traffic workloads.** * We moved the two largest Puma Agent sharded deployments back to ECS while we evaluate how the threading changes impact Redis connections and our autoscaling sensitivity. **Bound aggregate Redis connection demand.** * We are reviewing Redis connection-pool sizing, application thread counts, and workload scaling limits so that application growth cannot exceed the available Redis connection capacity.

Read the full incident report →

Started Sep 04, 2026, 01:42 AM UTC· resolved Sep 04, 2026, 02:39 AM UTC

1. investigating Sep 04, 2026, 01:42 AM UTC

   We're seeing increased latency and error rates within our Hosted Agents. We're currently investigating and will provide status updates as they become available.

2. monitoring Sep 04, 2026, 02:23 AM UTC

   We have deployed mitigations and have observed recovery across Hosted Agents. We will continue monitoring.

3. resolved Sep 04, 2026, 02:39 AM UTC

   Jobs running on Hosted Agents are now being dispatched promptly.

Read the full incident report →

Started Sep 03, 2026, 06:00 AM UTC· resolved Sep 03, 2026, 07:06 AM UTC

1. investigating Sep 03, 2026, 06:00 AM UTC

   We're seeing increased latency and error rates for a subset of our customers. We're currently investigating and will provide status updates as they become available.

2. investigating Sep 03, 2026, 06:09 AM UTC

   Some job dispatches to hosted agents are timing out, leading to jobs starting late. We're currently investigating and will update again soon.

3. monitoring Sep 03, 2026, 06:44 AM UTC

   We're seeing recovery in hosted job scheduling rates. Your jobs should run, but there may be some delays as we work through any backlogs.

4. resolved Sep 03, 2026, 07:06 AM UTC

   Jobs running on Hosted Agents are now being dispatched promptly.

Read the full incident report →

Started Sep 02, 2026, 02:45 PM UTC· resolved Sep 02, 2026, 03:51 PM UTC

1. investigating Sep 02, 2026, 02:45 PM UTC

   We're investigating an issue with job dispatch to our Hosted Agents.

2. investigating Sep 02, 2026, 03:16 PM UTC

   Our engineers are continuing to investigate an issue with delayed job dispatch for Hosted Agents. Job dispatch for self-hosted agents remains unaffected.

3. identified Sep 02, 2026, 03:29 PM UTC

   We're beginning to see recovery with delayed job dispatch for Hosted Agents. We are currently working through the backlog of delayed jobs.

4. monitoring Sep 02, 2026, 03:43 PM UTC

   We are continuing to process through the backlog of delayed jobs and monitoring.

5. resolved Sep 02, 2026, 03:51 PM UTC

   Job dispatch for Hosted Agents is no longer experiencing delays.

Read the full incident report →

Started Sep 01, 2026, 03:32 PM UTC· resolved Sep 01, 2026, 04:52 PM UTC

1. identified Sep 01, 2026, 03:32 PM UTC

   We are experiencing delivery issues with email notifications due to an issue with an upstream provider.

2. identified Sep 01, 2026, 04:19 PM UTC

   Our on-call engineers are working on failing over to an alternative upstream email provider to restore outbound email notifications. No other Buildkite services have been impacted by this disruption.

3. monitoring Sep 01, 2026, 04:38 PM UTC

   We are seeing successful email deliveries and are monitoring processing of our outbound email queue.

4. resolved Sep 01, 2026, 04:52 PM UTC

   Our outbound email queue has finished processing all previously failed deliveries. Email notifications are processing successfully.

Read the full incident report →

Started Aug 27, 2026, 10:56 AM UTC· resolved Aug 27, 2026, 11:47 AM UTC

1. monitoring Aug 27, 2026, 10:56 AM UTC

   Internal processing of uploaded test execution data has been delayed, and is currently catching up. No data has been dropped, and all data should be processed over the next hour.

2. monitoring Aug 27, 2026, 11:23 AM UTC

   Catch-up processing continues, most of the backlog is processed, ingestion latency is decreasing.

3. resolved Aug 27, 2026, 11:47 AM UTC

   Processing of uploaded test execution data has caught up and is back to normal.

Read the full incident report →

Started Aug 26, 2026, 01:42 AM UTC· resolved Aug 26, 2026, 03:13 AM UTC

1. investigating Aug 26, 2026, 01:42 AM UTC

   We're seeing increased latency and error rates for a subset of our customers. We're currently investigating and will provide status updates as they become available.

2. investigating Aug 26, 2026, 02:02 AM UTC

   We identified issues affecting certain customers when the buildkite-agent attempts to register with Buildkite. When agents are not able to register, builds may start to back up. We've identified a potential root case and are attempting changes to fix it.

3. investigating Aug 26, 2026, 02:36 AM UTC

   We have rolled out a few changes to fix the underlying issues and are seeing signs of recovery. We'll keep monitoring until all systems are fully recovered and communicate once we are confident everything is stable again.

4. monitoring Aug 26, 2026, 02:43 AM UTC

   Recovery is underway; we'll keep monitoring until recovery is complete.

5. resolved Aug 26, 2026, 03:13 AM UTC

   Mitigation strategies worked as expected and now agent registration is back to normal. All systems are stable now.

Read the full incident report →

Started Aug 25, 2026, 06:41 AM UTC· resolved Aug 25, 2026, 08:16 AM UTC

1. monitoring Aug 25, 2026, 06:41 AM UTC

   At 06:04 UTC we deployed a defective request routing change to `agent-edge.buildkite.com`, which caused some customers' agents to abruptly disconnect. At 06:09 UTC we noticed the problem, and immediately reverted the change. We began to observe recovery at 06:10 UTC, and are continuing to monitor.

2. monitoring Aug 25, 2026, 07:42 AM UTC

   Agent connectivity is recovered, but some agents are not reconnecting automatically after the earlier forced disconnects. We recommend restarting agent service to make agents reconnect.

3. resolved Aug 25, 2026, 08:16 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 13, 2026, 03:41 PM UTC· resolved Aug 13, 2026, 04:40 PM UTC

1. investigating Aug 13, 2026, 03:41 PM UTC

   We've spotted that something has gone wrong and there's elevated latency. We're currently investigating the issue, and will provide an update soon.

2. identified Aug 13, 2026, 04:14 PM UTC

   We've identified the cause of the issue and are actively mitigating.

3. monitoring Aug 13, 2026, 04:29 PM UTC

   We have identified and fixed the issue. We are monitoring and seeing signs of improvement.

4. resolved Aug 13, 2026, 04:40 PM UTC

   We think the impact from the issue is over.

Read the full incident report →

Started Aug 05, 2026, 10:00 PM UTC· resolved Aug 05, 2026, 10:16 PM UTC

1. resolved Aug 05, 2026, 11:13 PM UTC

   Between 22:03 - 22:16 UTC customers experienced delayed job dispatch, build processing and notifications. Additionally there were errors receiving webhooks during this period.

Read the full incident report →

Started Aug 03, 2026, 05:11 AM UTC· resolved Aug 03, 2026, 05:36 AM UTC

1. investigating Aug 03, 2026, 05:11 AM UTC

   We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.

2. identified Aug 03, 2026, 05:17 AM UTC

   We're experiencing latency with processing job dispatch, pipeline uploads and incoming webhooks. We have identified the cause and are rolling forward with a fix now.

3. identified Aug 03, 2026, 05:22 AM UTC

   We've identified that the impact is more widespread that we initially understood - scheduled builds as well as job dispatch. The fix that is rolling out now is expected to address all impact.

4. monitoring Aug 03, 2026, 05:29 AM UTC

   The fix has been fully deployed and we are seeing signs of recovery. All events impacted by this issue will be automatically retried. We are continuing to monitor the fix for stability.

5. resolved Aug 03, 2026, 05:36 AM UTC

   We've confirmed all impact from this incident has been resolved.

6. postmortem Aug 10, 2026, 12:03 AM UTC

   ## Service Impact Between 04:40 and 05:34 UTC on 3 August 2026, Buildkite stopped running recurring background work. Every background job process across the product was affected. The customer-visible impact was concentrated in Pipelines. During this window, scheduled builds did not start at their scheduled time, jobs waited longer than usual before being dispatched to an agent, in some cases by more than ten minutes, and pipeline uploads and inbound webhook processing were delayed. Job timeouts were not enforced, so a job that should have timed out kept running, and agents that had finished or gone away were not cleaned up promptly. Scheduled builds were not skipped. Every schedule that came due during the window ran once background scheduling recovered, up to around forty minutes late. A schedule that fires more often than that ran once on recovery rather than once for each interval it missed. No builds, jobs or queued work were lost. Work that was delayed was processed once the incident was resolved, and jobs that errored were retried automatically. The dashboard, the REST and GraphQL APIs, and agents already running jobs were not affected. ## Incident Summary At 04:33 UTC we merged a routine update to Rails, the web framework Buildkite runs on, taken to stay current with a security patch. Alongside Rails, that update pulled `connection_pool` from 2.5.5 to 3.0.2. The 3.0 release made that library's API keyword-only, which changes how every caller has to invoke it. Our review of the change covered our own code's use of `connection_pool` and updated it accordingly. It did not cover the other gems in our bundle that call the same library. One of those is Sidekiq, which runs our background jobs. From 04:40 UTC, as the deploy rolled out, three internal Sidekiq threads inside every worker process failed on startup and stopped. The processes themselves stayed healthy and kept working jobs that were already queued, so nothing crashed and no deploy failed. What stopped was the scheduling of recurring work, which is what triggers scheduled builds, dispatches queued jobs to agents, enforces job timeouts and cleans up finished agents. That scheduling currently runs through a single elected process for the whole background job fleet, so it stopped for every product and every customer at the same moment. Our monitoring alerted us at 04:48 UTC and we identified the change responsible at 04:59. Our first assessment was that there was no customer impact, because everything we could see was still moving. It took another ten minutes to recognise that recurring work had stopped across the whole product, at which point we published a status page and raised the severity. A fix pinning `connection_pool` back to 2.5.5 was merged at 05:10 and deployed ahead of the normal merge queue. Scheduling resumed at around 05:19 and the incident was resolved at 05:34. ## What contributed to this happening `connection_pool` is called both by our own code and by gems we depend on. Our review, and the automated review tooling that flagged the change on the pull request, both looked at how we call it and stopped there. Nothing in the dependency graph prevented the combination. Sidekiq 7.3.9, the version we run, permits any version of `connection_pool` from 2.3.0 upwards. Sidekiq 7.3.10 added an upper bound that would have refused 3.0, but we had not adopted it. The failure only appears when a real Sidekiq process starts up. Our test suite does not exercise that path, and our pre-production check verifies that the application boots rather than that background scheduling is still running. Because recurring work is scheduled by a single elected process for the whole fleet, one failure removed it for every customer at once rather than for a single group of customers. ## Changes we're making We have pinned `connection_pool` to a version compatible with the Sidekiq release we run. This shipped as part of resolving the incident. We are upgrading Sidekiq to 7.3.10, which declares an upper bound on `connection_pool` and so makes this combination impossible to resolve rather than something we have to remember to avoid. We are changing how recurring work is scheduled so that it is no longer driven by a single process for the whole fleet. Each group of shards will schedule its own recurring work independently. This would not by itself have prevented this incident, because the incompatible library was loaded by every process, but it removes a shared point of failure that can currently stop recurring work for every customer at once. This work is already in progress. We are updating our internal engineering guidance so that an upgrade moving a shared library to a new major version is reviewed against every consumer of that library, including the other libraries that depend on it, rather than only our own code.

Read the full incident report →

Started Jun 11, 2026, 11:18 PM UTC· resolved Jun 12, 2026, 12:16 AM UTC

1. investigating Jun 11, 2026, 11:18 PM UTC

   We're observing increased latency for all our customers. We're currently investigating and will provide status updates as they become available.

2. monitoring Jun 11, 2026, 11:49 PM UTC

   We've isolated the issue to elevated load on our REST API service and are working to mitigate. The agent and stacks API isn’t affected.

3. monitoring Jun 11, 2026, 11:51 PM UTC

   We've isolated the issue to elevated load on our REST API service and have mitigated the issue. The agent and stacks API isn’t affected.

4. resolved Jun 12, 2026, 12:16 AM UTC

   The mitigation applied before the last update had the intended effect, and we have seen recovery in REST API latency.

Read the full incident report →

Started Jun 11, 2026, 12:32 AM UTC· resolved Jun 11, 2026, 12:51 AM UTC

1. investigating Jun 11, 2026, 12:32 AM UTC

   We're observing increased latency and error rates for a subset of our customers on the Agent API. We're currently investigating and will provide status updates as they become available.

2. resolved Jun 11, 2026, 12:51 AM UTC

   Between 00:05 - 00:34 UTC, a subset of customers experienced increased latency and timeout errors on the Agent API. This impacts job assignment. At peak impact, we saw an error rate of 1.3% of requests and job acceptance latency up to 53s.

Read the full incident report →

Started May 30, 2026, 12:30 AM UTC· resolved May 30, 2026, 12:30 AM UTC

1. resolved May 31, 2026, 11:45 PM UTC

   We have received reports email deliveries have not been working, affecting signup and invite emails as well as build notification emails. This issue has now been resolved.

Read the full incident report →

Started May 15, 2026, 06:51 AM UTC· resolved May 15, 2026, 07:35 AM UTC

1. monitoring May 15, 2026, 06:51 AM UTC

   Ingestion of Test Engine execution data from an internal queue to a data store stalled, has been resumed, and is working through the backlog. Visibility of test executions from the past hour hours will be delayed for approximately a further one hour. This has been a recurring issue; an architectural change is coming soon to eliminate this failure mode.

2. resolved May 15, 2026, 07:35 AM UTC

   Processing of the backlog is complete.

Read the full incident report →