Wasabi incident

System Errors in AP-NORTHEAST-1

Resolved after 14 hours 45 minutes, on March 24, 2026.

postmortem Apr 27, 2026, 11:38 PM UTC

Between 24 March 2026 at 19:30 UTC and 25 March 2026 at 10:15 UTC, Wasabi S3 read operations in the Tokyo region \(ap-northeast-1\) were impacted for a small subset of customers and buckets. The impact was caused by a software deployment that unintentionally impacted one of our storage pools, resulting in the inability to access data stored on that pool. No other storage pools in the Tokyo region were affected. Once this was detected the pool was enabled again and data access restored. Procedural changes were added and active monitoring was already added to avoid this specific type of incident in the future.

Started
Mar 24, 2026, 07:30 PM UTC
Resolved
Mar 25, 2026, 10:15 AM UTC
Duration
14h 45m
Detected by Pingoru
Mar 24, 2026, 07:30 PM UTC

Update timeline

  1. resolved Mar 24, 2026, 07:30 PM UTC

    From 3/24 19:30 UTC to 3/25 10:15 UTC customers may have experienced elevated HTTP 5XX error responses when interacting with their Wasabi bucket(s) is AP-NORTHEAST-1 region. This issue has been resolved. For any questions please reach out to [email protected].

  2. postmortem Apr 27, 2026, 11:38 PM UTC

    Between 24 March 2026 at 19:30 UTC and 25 March 2026 at 10:15 UTC, Wasabi S3 read operations in the Tokyo region \(ap-northeast-1\) were impacted for a small subset of customers and buckets. The impact was caused by a software deployment that unintentionally impacted one of our storage pools, resulting in the inability to access data stored on that pool. No other storage pools in the Tokyo region were affected. Once this was detected the pool was enabled again and data access restored. Procedural changes were added and active monitoring was already added to avoid this specific type of incident in the future.

Your product runs on more than Wasabi

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →