Wasabi Account Control Manager Console views for Control Accounts, Sub-Accounts and Invoices not loading

Update timeline

1. investigating Oct 30, 2024, 11:14 PM UTC

   We are currently investigating this issue and expect to have these views restored shortly.

2. investigating Oct 31, 2024, 12:34 AM UTC

   While some of the views have been restored, we continue to work to resolve the issue. This page will be updated as more progress is made.

3. identified Oct 31, 2024, 12:44 AM UTC

   The issue has been identified and a fix is being implemented.

4. resolved Oct 31, 2024, 09:55 AM UTC

   This incident has been resolved.

5. postmortem Nov 07, 2024, 03:24 PM UTC

   From 2024-10-31 00:42 UTC to 2024-10-31 10:03 UTC, we experienced an issue where Control Accounts, Sub-Accounts and Invoice data in WACM was not loading properly when attempted to be viewed by the client. Upon investigation by our Engineering Team, it was found that the nodes responsible for fetching and delivering this data were in a bad state. Once the nodes were restored to a functional state, our team started the process of re-indexing the data, which would allow the data to be served via the WACM portal successfully. At 10:03 UTC on October 31, 2024, the WACM data was fully available to customers.