Nanonets incident
Delayed file processing for models using async api in EU region
Resolved after 4 hours 48 minutes, affecting API, on January 27, 2026.
**Incident Summary** On 27th Jan 17:00 UTC, we experienced elevated load on one of our core LLM processing engines due to a sudden increase in traffic volume. This primarily affected file processing workflows in the **EU region**. **Impact** * **Synchronous APIs** may have experienced intermittent failures or increased error rates. * **Asynchronous file processing** was degraded, with increased processing times. * Files submitted during this period were not lost and were eventually processed once the backlog was cleared, though with higher-than-usual latency. **Root Cause** A surge in traffic led to sustained high load on a core LLM engine, resulting in slower processing and queue buildup for async workflows in the EU region. **Resolution** Our engineering team actively monitored the situation and added additional processing capacity to handle the increased load and clear the backlog. Due to the size of the queue, backlog clearance took several hours, during which async response times were elevated. Once capacity was scaled and the backlog was processed, services returned to normal operation. We sincerely apologize for the inconvenience caused by this incident. We are working on adding more capacity on-demand so that this issue wont repeat again. Thank you for your understanding.
Affected components
Update timeline
- investigating Jan 27, 2026, 05:24 PM UTC
We are currently investigating this issue.
- identified Jan 27, 2026, 06:27 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Jan 27, 2026, 09:32 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jan 27, 2026, 10:12 PM UTC
This incident has been resolved.
- postmortem Jan 30, 2026, 05:56 AM UTC
**Incident Summary** On 27th Jan 17:00 UTC, we experienced elevated load on one of our core LLM processing engines due to a sudden increase in traffic volume. This primarily affected file processing workflows in the **EU region**. **Impact** * **Synchronous APIs** may have experienced intermittent failures or increased error rates. * **Asynchronous file processing** was degraded, with increased processing times. * Files submitted during this period were not lost and were eventually processed once the backlog was cleared, though with higher-than-usual latency. **Root Cause** A surge in traffic led to sustained high load on a core LLM engine, resulting in slower processing and queue buildup for async workflows in the EU region. **Resolution** Our engineering team actively monitored the situation and added additional processing capacity to handle the increased load and clear the backlog. Due to the size of the queue, backlog clearance took several hours, during which async response times were elevated. Once capacity was scaled and the backlog was processed, services returned to normal operation. We sincerely apologize for the inconvenience caused by this incident. We are working on adding more capacity on-demand so that this issue wont repeat again. Thank you for your understanding.
Your product runs on more than Nanonets
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services