Pokémon Vortex incident

Elevated database connection failures

Resolved after 4 days 17 hours, affecting Game Web Servers and Database, on November 9, 2023.

monitoring Nov 14, 2023, 04:23 AM UTC

The game is now up and running and we are monitoring the new server's stability.

Started
Nov 09, 2023, 11:28 AM UTC
Resolved
Nov 14, 2023, 05:15 AM UTC
Duration
4d 17h
Detected by Pingoru
Nov 09, 2023, 11:28 AM UTC

Affected components

Game Web ServersDatabase

Update timeline

  1. investigating Nov 09, 2023, 11:28 AM UTC

    We are currently investigating a high level of database connection failures.

  2. investigating Nov 09, 2023, 03:35 PM UTC

    One or more hardware components in the database server is failing, causing a huge slowdown in read/write processes. We are currently in the process of retrieving all files from the server which is an incredibly slow process due to this issue and based on the current speed, it is likely to take days. Unfortunately, during this time, the game will be unplayable and you will not be able to interact with the Discord bot. Any updates on this will be posted here if/when changes occur.

  3. investigating Nov 12, 2023, 04:32 PM UTC

    All data has successfully been retrieved from the server and we are now investigating the cause of the problem to have the necessary hardware repaired or replaced.

  4. identified Nov 12, 2023, 11:11 PM UTC

    The primary SSD of the server is confirmed to be failing. Rather than replace it, we will take the opportunity to upgrade the server entirely, so we are currently in talks with the datacentre in Switzerland to have a new server provisioned.

  5. identified Nov 13, 2023, 01:51 PM UTC

    Provisioning of the server is complete. We are now in the process of setting up the new server with the necessary software and configuration.

  6. identified Nov 13, 2023, 05:07 PM UTC

    Server setup is now complete and we are currently in the process up transferring all data and game files to the new server.

  7. monitoring Nov 14, 2023, 01:46 AM UTC

    All data has been transferred. We will now test everything to make sure it all runs smoothly and set up an event for Arceus (Psychic). An announcement will follow on our Discord when the game is back up to play, hopefully it won't be much longer now. https://discord.gg/pokemonvortex

  8. monitoring Nov 14, 2023, 04:23 AM UTC

    The game is now up and running and we are monitoring the new server's stability.

  9. resolved Nov 14, 2023, 05:15 AM UTC

    This incident has been resolved.

Your product runs on more than Pokémon Vortex

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →