Siena incident

Backfill Service Unavailability

Resolved on January 1, 2025, affecting Backfill Service.

postmortem Jan 14, 2025, 10:26 AM UTC

# Impact: Shopify webhooks were disabled due to repeated failures in returning 200-status responses during downtime, affecting 11 Shopify integrations. The backfill service, which compensates for missing data, was also down, causing major disruptions in data synchronization for orders, customers, and updates. # Timeline: * Dec 17: Backfill service stopped working * Dec 23 - 31: Multiple tickets raised for missing data. * Dec 31: Webhooks restored Jan 1: Backfill service restored & executed

Started
Jan 01, 2025, 12:00 AM UTC
Resolved
Jan 01, 2025, 12:00 AM UTC
Duration
–
Detected by Pingoru
Jan 01, 2025, 12:00 AM UTC

Affected components

Backfill Service

Update timeline

  1. resolved Jan 14, 2025, 10:26 AM UTC

    We experienced an issue with the Backfill Service, which ensures synchronisation of Shopify data in cases where webhooks are missing. During this time, orders for Shopify integrations were not available in our system, specifically affecting integrations where webhooks were disabled due to repeated non-200 responses.

  2. postmortem Jan 14, 2025, 10:26 AM UTC

    # Impact: Shopify webhooks were disabled due to repeated failures in returning 200-status responses during downtime, affecting 11 Shopify integrations. The backfill service, which compensates for missing data, was also down, causing major disruptions in data synchronization for orders, customers, and updates. # Timeline: * Dec 17: Backfill service stopped working * Dec 23 - 31: Multiple tickets raised for missing data. * Dec 31: Webhooks restored Jan 1: Backfill service restored & executed

Your product runs on more than Siena

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →