An increase in errors and delays from our services

Update timeline

1. investigating Dec 19, 2024, 05:01 PM UTC

   We are currently investigating this incident..

2. investigating Dec 19, 2024, 06:01 PM UTC

   We have reverted the most recent release, and the error rates have gone down. We are still experiencing some slowdowns due to the task backlog that grew during the incident, but the system is mostly recovered. Runs that were created during the outage will be marked as timed out, and you may see errors related to those runs in your local clients if they are still in progress. New runs should not be affected, and should now work as expected. We are still actively investigating the root cause..

3. identified Dec 19, 2024, 06:23 PM UTC

   We are still seeing issues as a result of the backlog, we are scaling up to address the backlog of tasks. We are continuing to investigate the cause in the meantime..

4. monitoring Dec 19, 2024, 07:05 PM UTC

   Our backlog has recovered, and the service is back to normal. The root cause appears to have been changes in our now reverted deployment. We will continue to investigate the cause and provide more details after our investigation..

5. resolved Dec 19, 2024, 08:02 PM UTC

   The system has recovered. We will continue to investigate the root cause and will update the description of the incident with the details when we have them..