DataRobot incident

Batch Jobs queued up in US MTS

Resolved after 5 days, affecting Website and API and 1 more, on May 29, 2026.

resolved Jun 03, 2026, 09:29 AM UTC

A fix has been deployed to production and the issue is now resolved. All systems are operating normally.

Started
May 29, 2026, 08:37 AM UTC
Resolved
Jun 03, 2026, 09:29 AM UTC
Duration
5d
Detected by Pingoru
May 29, 2026, 08:37 AM UTC

Affected components

WebsiteAPIPredictionsAutoMLAI Catalog and Data IngestAI AppsMLOpsPipelineNotebooksGenerative AI LLM Playground

Update timeline

  1. identified May 29, 2026, 08:37 AM UTC

    Batch jobs that are currently in the queue are completed successfully; however, their completion status is not being updated correctly. As a temporary workaround, we are manually marking these jobs as completed while we work on implementing a solution.

  2. identified May 29, 2026, 10:22 AM UTC

    The affected jobs have been resolved and the issue is mitigated. Engineering is actively working on a permanent fix and testing is currently underway

  3. monitoring May 29, 2026, 02:38 PM UTC

    The remediation script has been deployed and Engineering is actively monitoring the situation. Batch jobs may take longer than usual to show as completed until a permanent fix is rolled out in the next production deployment.

  4. monitoring May 29, 2026, 03:20 PM UTC

    We are continuing to monitor for any further issues.

  5. resolved Jun 03, 2026, 09:29 AM UTC

    A fix has been deployed to production and the issue is now resolved. All systems are operating normally.

Your product runs on more than DataRobot

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →