Wasabi incident

System Errors to access Wasabi services

Resolved after 5 minutes, affecting US-Central-1 (Texas) and US-East-1 (N. Virginia) and 1 more, on March 27, 2025.

postmortem Apr 03, 2025, 09:10 PM UTC

On 27 March 2025 from 10:09 UTC to approximately 10:14 UTC, Wasabi services in our Ashburn \(us-east-1\), Manassas \(us-east-2\), Hillsboro \(us-west-1\), Plano \(us-central-1\), Amsterdam \(eu-central-1\), Tokyo \(ap-northeast-1\), Osaka \(ap-northeast-2\), London \(eu-west-1\), and Frankfurt \(eu-central-2\) regions were unavailable due to an issue, impacting customer traffic and Web Console connectivity. Wasabi Engineering and Operations teams were notified by our automated monitoring systems of an issue with a spike in connection between internal service worker nodes. This spike led to the inability to open new connections within the services. Wasabi’s Operations Team manually restarted the services, which corrected the issue and allowed for client connections to resume successfully. By 10:14 UTC, all internal services were restarted and all services returned to normal operating conditions.

Started
Mar 27, 2025, 10:09 AM UTC
Resolved
Mar 27, 2025, 10:14 AM UTC
Duration
5m
Detected by Pingoru
Mar 27, 2025, 10:09 AM UTC

Affected components

US-Central-1 (Texas)US-East-1 (N. Virginia)US-East-2 (N. Virginia)US-West-1 (Oregon)EU-Central-1 (Amsterdam)EU-Central-2 (Frankfurt)EU-West-2 (Paris)AP-Northeast-1 (Tokyo)AP-Northeast-2 (Osaka)Wasabi Management Console

Update timeline

  1. resolved Mar 28, 2025, 01:53 AM UTC

    Between 10:09PM to 10:14PM UTC, we experienced errors with connecting to the Wasabi Web Console, STS, and IAM services. It was a temporary issue, auto-resolved and Operations Team validated it. There is no longer any further impact, and all services are now running without issue.

  2. postmortem Apr 03, 2025, 09:10 PM UTC

    On 27 March 2025 from 10:09 UTC to approximately 10:14 UTC, Wasabi services in our Ashburn \(us-east-1\), Manassas \(us-east-2\), Hillsboro \(us-west-1\), Plano \(us-central-1\), Amsterdam \(eu-central-1\), Tokyo \(ap-northeast-1\), Osaka \(ap-northeast-2\), London \(eu-west-1\), and Frankfurt \(eu-central-2\) regions were unavailable due to an issue, impacting customer traffic and Web Console connectivity. Wasabi Engineering and Operations teams were notified by our automated monitoring systems of an issue with a spike in connection between internal service worker nodes. This spike led to the inability to open new connections within the services. Wasabi’s Operations Team manually restarted the services, which corrected the issue and allowed for client connections to resume successfully. By 10:14 UTC, all internal services were restarted and all services returned to normal operating conditions.

Your product runs on more than Wasabi

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →