Is Sauce Labs down?

We are seeing elevated error rates for Real Device tests in EU-Central-1 data center. We are investigating.

The root cause of the elevated error rates for Real Device tests in the EU-Central-1 data center has been identified. Remediation has been applied, and we are actively monitoring the situation.

After taking remedial action, Real Device test error rates have returned to normal in the EU-Central-1 data center. All services are fully operational.

We are currently seeing elevated error rates for macOS and iOS tests in the US-West-1 data center. We are currently investigating.

After taking remedial action, all macOS and iOS tests are now performing as expected in the US-West-1 Data center. All services are fully operational.

### **Dates:** Tuesday August 25th 2026, 11:38 – 13:10 UTC ### **What happened:** Customers running macOS and iOS tests in our US-West region experienced degraded service. Roughly 50% of the virtual Mac capacity in the region stopped accepting new tests, so tests either queued or failed to start. Remaining capacity came under additional pressure as work shifted onto it, which extended start times for both desktop and simulator tests. ### **Why it happened:** An internal security certificate used by our Mac hosts to reach a supporting cloud service reached its expiry date. Once it lapsed, the hosts could no longer establish a trusted connection to that service and stopped provisioning new test machines. The certificate had been issued manually and had neither automated renewal nor expiry alerting. ### **How we fixed it:** We issued and deployed a replacement certificate, which restored connectivity and returned Mac capacity to normal levels. ### **What we are doing to prevent it from happening again:** We are moving these certificates onto automated renewal and adding alerting so they are replaced well ahead of expiry, along with reviewing the surrounding tooling to make certificate handling safer.

We are currently seeing elevated error rates for virtual desktop tests in the US-West-1 datacenter. We are currently investigating.

After taking remedial action, Virtual Desktop tests are starting successfully in the US-West-1 Data center. All services are fully operational. We are closely monitoring the situation

### **Dates:** Friday, August 7th 2026, 11:00 UTC - 13:04 UTC. ### **What happened:** Windows and Intel Mac jobs in `us-west1` failed to start due to virtual machine \(VM\) allocation starvation. ### **Why it happened:** A service crash loop left VMs in an allocated but unclaimed state, while a cleanup bug prevented the system from releasing the orphaned capacity to boot new VMs. ### **How we fixed it:** Restored service stability and cleared stale allocations to resume VM provisioning and clear queued jobs. ### **What we are doing to prevent it from happening again:** Fixing allocator cleanup logic, strengthening deployment health checks, and improving capacity accounting for stale allocations.

Between July 16th 13:03 UTC and July 16th 17:33 UTC, we experienced a service disruption where symbol archives utilizing multi-part upload protocols failed to process. Incident has been resolved and all services are operational.

### **Dates:** Thursday July 16th 2026, 13:03 – 17:33 UTC ### **What happened:** Symbol archives uploaded in multiple parts failed to process. All regions were affected. ### **Why it happened:** Our symbol processing service sent an upload-verification field that our cloud storage provider's API does not accept for multi-part uploads, so those uploads were rejected. A fix for this had already been developed, but it had not yet been included in a released build and the service was not configured to use it. As in the first incident, rejected uploads were retried and accumulated on local disk. ### **How we fixed it:** We deployed a build containing the fix and corrected the service configuration on the affected workers. A large backlog of uploads then processed, which briefly re-filled the disks before draining completely. ### **What we are doing to prevent it from happening again:** We are releasing the fix formally through our build pipeline and persisting the corrected configuration in our configuration management, so it can't be lost. The disk-utilization alerting added after the first incident also covers the disk-exhaustion pattern common to both.

Between July 14th 23:45 UTC and July 15th 23:05 UTC, we experienced an issue where symbol archive uploads to Backtrace projects failed with HTTP 400 errors. Incident has been resolved and all services are operational.

### **Dates:** Tuesday July 14th 2026, 23:45 UTC – Wednesday July 15th 2026, 23:05 UTC ### **What happened:** Symbol archive uploads to Error Reporting \(Backtrace\) projects failed with HTTP 400 errors. All regions were affected. ### **Why it happened:** The credentials our symbol processing service used to write to cloud storage were no longer valid, so uploads could not be stored. Failed uploads were retried repeatedly and accumulated on local disk until the service ran out of space, at which point it also began rejecting new uploads. ### **How we fixed it:** We reissued the storage credentials, increased disk capacity on the affected workers, and restarted the service. The queued uploads then processed successfully, and we confirmed recovery with affected customers. ### **What we are doing to prevent it from happening again:** We've added disk-utilization monitoring and alerting to this service so we detect the condition ourselves before it affects uploads, rather than relying on customer reports.