Nanonets incident

High Response Times for Non Instant learning models in US region

Resolved after 48 minutes, affecting API, on February 21, 2025.

postmortem Feb 21, 2025, 03:27 PM UTC

On 21st Feb 2025, from 6:00 PM IST to 08:30 PM IST, non-instant learning models on [app.nanonets.com](http://app.nanonets.com) experienced high response times and timeouts due to extreme load on our system. Although auto-scaling was triggered on time, one of our services choked under the increased throughput. Our engineers quickly identified the bottleneck and increased the service's throughput, normalizing the response times. We apologize for the inconvenience and we are implementing measures to enhance system resilience and better handle future traffic spikes.

Started
Feb 21, 2025, 02:19 PM UTC
Resolved
Feb 21, 2025, 03:07 PM UTC
Duration
48m
Detected by Pingoru
Feb 21, 2025, 02:19 PM UTC

Affected components

API

Update timeline

  1. investigating Feb 21, 2025, 02:19 PM UTC

    We are currently investigating this issue.

  2. monitoring Feb 21, 2025, 03:04 PM UTC

    A fix has been implemented and we are monitoring the results.

  3. monitoring Feb 21, 2025, 03:04 PM UTC

    We are continuing to monitor for any further issues.

  4. resolved Feb 21, 2025, 03:07 PM UTC

    This incident has been resolved.

  5. postmortem Feb 21, 2025, 03:27 PM UTC

    On 21st Feb 2025, from 6:00 PM IST to 08:30 PM IST, non-instant learning models on [app.nanonets.com](http://app.nanonets.com) experienced high response times and timeouts due to extreme load on our system. Although auto-scaling was triggered on time, one of our services choked under the increased throughput. Our engineers quickly identified the bottleneck and increased the service's throughput, normalizing the response times. We apologize for the inconvenience and we are implementing measures to enhance system resilience and better handle future traffic spikes.

Your product runs on more than Nanonets

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →