Nacelle incident

Degraded Storefront API Performance

Resolved on November 11, 2023.

resolved Nov 13, 2023, 09:57 PM UTC

Between 6:26 PM ET and 7:21 PM ET, Nacelle experienced a spike of new requests that required Nacelle’s DevOps team to scale additional Kubernetes pods. While these new pods were scaling up, the API experienced degraded performance (e.g., long latencies). The root cause of the issue was how Nacelle's custom schema fetching works across new additions to our GraphQL interface. To ensure stability over Black Friday / Cyber Monday, the Nacelle team has removed features still in development from the Storefront API schema service.

Started
Nov 11, 2023, 11:30 PM UTC
Resolved
Nov 11, 2023, 11:30 PM UTC
Duration
–
Detected by Pingoru
Nov 11, 2023, 11:30 PM UTC

Update timeline

  1. resolved Nov 13, 2023, 09:57 PM UTC

    Between 6:26 PM ET and 7:21 PM ET, Nacelle experienced a spike of new requests that required Nacelle’s DevOps team to scale additional Kubernetes pods. While these new pods were scaling up, the API experienced degraded performance (e.g., long latencies). The root cause of the issue was how Nacelle's custom schema fetching works across new additions to our GraphQL interface. To ensure stability over Black Friday / Cyber Monday, the Nacelle team has removed features still in development from the Storefront API schema service.

Your product runs on more than Nacelle

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →