Database outage

Update timeline

1. identified May 21, 2025, 09:02 PM UTC

   At approximately 16:30 UTC, an unusually high volume of specific write queries from our application began to strain our main database. This load overwhelmed our internal data replication system. As a result, a backlog of data awaiting replication quickly grew, consuming all available space on the main database's system disk and forcing it offline. We are currently restoring the database from a snapshot; however, this process is taking longer than expected because the snapshot includes this large volume of non-replicated data, which is inherently slower to process during recovery.

2. identified May 21, 2025, 09:12 PM UTC

   The data restoration process is nearly complete. We estimate that the system will be fully back to normal operation within the next hour. Thank you for your patience as we bring services back online.

3. resolved May 21, 2025, 10:58 PM UTC

   Our systems have fully recovered and are back to normal operation. The data restoration process has completed successfully, and all services—API and Phantoms—are now functioning as expected. We’ve addressed the issue caused by an unusually high volume of write queries that led to a backlog in our internal replication system. We sincerely thank you for your patience throughout the resolution process. If you continue to experience any issues, please don’t hesitate to reach out to our support team.