Resolved after 3 hours 32 minutes, affecting Dashboard and API, on February 27, 2025.
Between 11:45 a.m. and 12:20 p.m. PST, a batch request began to loop unexpectedly, interfering with each other and causing container contention. Because of how batch creations are processed, this prevented other jobs from being processed until they were cleared manually. Once the job was manually stopped, the system returned to normal batch processing. The team will implement enhanced monitoring to avoid this scenario in the future.
Affected components
Update timeline
- investigating Feb 27, 2025, 09:41 PM UTC
At this time the internal process for managing task creation queues for the asynchronous batch task creation endpoint is experiencing issues. This is being investigated with high urgency.
- monitoring Feb 28, 2025, 12:38 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Feb 28, 2025, 01:13 AM UTC
This incident has been resolved.
- postmortem Mar 17, 2025, 03:18 PM UTC
Between 11:45 a.m. and 12:20 p.m. PST, a batch request began to loop unexpectedly, interfering with each other and causing container contention. Because of how batch creations are processed, this prevented other jobs from being processed until they were cleared manually. Once the job was manually stopped, the system returned to normal batch processing. The team will implement enhanced monitoring to avoid this scenario in the future.
Your product runs on more than Onfleet
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services