Elevated errors during document processing

Update timeline

1. investigating Jun 24, 2025, 06:35 PM UTC

   We are experiencing an increased error rate in document processing requests. A small percentage of requests are returning a 504 error response. We are investigating the issue.

2. investigating Jun 24, 2025, 09:37 PM UTC

   We have reduced error rates further by over provisioning compute resources and spreading requests across more worker nodes. We are continuing work on identifying the root cause of the issue.

3. investigating Jun 25, 2025, 12:36 AM UTC

   We have identified the root cause as being a newly surfaced edge case scenario in requests being submitted for processing. These particular requests consume an unusually high level of compute resources, which has downstream impact on other requests processed by the same worker processes. We are currently working on a solution to handle these edge cases gracefully.

4. resolved Jun 25, 2026, 01:45 AM UTC

   We have implemented and validated a fix for the problematic edge case. All systems are back to normal.