Spacelift incident

Delayed handling of events

Resolved after 1 hour 12 minutes, affecting Event processing and Public workers, on October 30, 2024.

resolved Oct 30, 2024, 04:53 PM UTC

Our monitors are reporting that all events have been processed. The issue has been resolved.

Started
Oct 30, 2024, 03:40 PM UTC
Resolved
Oct 30, 2024, 04:53 PM UTC
Duration
1h 12m
Detected by Pingoru
Oct 30, 2024, 03:40 PM UTC

Affected components

Event processingPublic workers

Update timeline

  1. investigating Oct 30, 2024, 03:40 PM UTC

    We are seeing an increased error rate when handling webhooks for all customers. This is causing a delay in runs being scheduled.

  2. monitoring Oct 30, 2024, 04:13 PM UTC

    We have introduced a quick fix for handling events.

  3. identified Oct 30, 2024, 04:28 PM UTC

    We have identified an issue with our object storage abstraction, we are rolling back to a previous release.

  4. resolved Oct 30, 2024, 04:53 PM UTC

    Our monitors are reporting that all events have been processed. The issue has been resolved.

Your product runs on more than Spacelift

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →