Sirv incident

Some CDN requests failing in California

Resolved after 44 minutes, affecting section-157198 (CDN request from Los Angeles), on March 31, 2025.

resolved Mar 31, 2025, 03:16 PM UTC

The issue has been resolved. The issue was caused by packet loss affecting our new Los Angeles datacenter. The datacenter traced the issue to their upstream provider, Arelion, which was experiencing network instability. The issue was resolved when Arelion was temporarily disabled as an upstream provider and traffic was rerouted through other providers. We require that our datacenters identify and reroute traffic automatically and quickly in such scenarios. However, the resolution of this issue was not fast enough and given that it occurred soon after adopting this new datacenter, we have decided to cease using it. All traffic is now being routed from our original Los Angeles datacenter (as of 7 April 2025).

Started
Mar 31, 2025, 02:32 PM UTC
Resolved
Mar 31, 2025, 03:16 PM UTC
Duration
44m
Detected by Pingoru
Mar 31, 2025, 02:32 PM UTC

Affected components

section-157198 (CDN request from Los Angeles)

Update timeline

  1. investigating Mar 31, 2025, 02:32 PM UTC

    Some requests from California are slow and a small number (under 2%) are failing. This is being investigated and likely related to new CDN servers that were deployed today in Los Angeles.

  2. resolved Mar 31, 2025, 03:16 PM UTC

    The issue has been resolved. The issue was caused by packet loss affecting our new Los Angeles datacenter. The datacenter traced the issue to their upstream provider, Arelion, which was experiencing network instability. The issue was resolved when Arelion was temporarily disabled as an upstream provider and traffic was rerouted through other providers. We require that our datacenters identify and reroute traffic automatically and quickly in such scenarios. However, the resolution of this issue was not fast enough and given that it occurred soon after adopting this new datacenter, we have decided to cease using it. All traffic is now being routed from our original Los Angeles datacenter (as of 7 April 2025).

Your product runs on more than Sirv

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →