Generating PDF sheet music is failing intermittently in SongSelect and Partner API

Update timeline

1. investigating Feb 10, 2025, 04:30 AM UTC

   We are currently investigating this incident..

2. identified Feb 10, 2025, 07:11 AM UTC

   We have identified the problem area and are working on a fix for this incident..

3. monitoring Feb 10, 2025, 02:01 PM UTC

   We implemented a fix and are currently monitoring the result..

4. monitoring Feb 10, 2025, 02:25 PM UTC

   After significant investigation, it appears the partner API was NOT negatively affected by this. It only affected our SongSelect application..

5. identified Feb 10, 2025, 03:24 PM UTC

   Althought the partner API was not affected from the prior issue, we are now seeing some new issues with the partner API. We are working on it..

6. monitoring Feb 10, 2025, 04:29 PM UTC

   At the moment, the system appears to be operating normal again. We will continue to monitor..

7. resolved Feb 10, 2025, 04:50 PM UTC

   We believe this incident is resolved, but we will continue to monitor over the coming days..

8. identified Feb 10, 2025, 07:06 PM UTC

   As we are monitoring, the problem has returned so we are reopening this incident. We are working on a solution. This is now likely also affecting the ability for Partners to get chord sheets, and it is impacting SongSelect ability to view Lead and Vocal sheets. .

9. monitoring Feb 10, 2025, 09:56 PM UTC

   Things seem to have stabilized since 11:20am Pacific Time and things have been operational since then. We are continuing to monitor closely..

10. resolved Feb 11, 2025, 03:16 PM UTC

    The system is stable. This incident has been marked resolved, but we will continue to monitor closely over the coming weeks..