Blacksmith incident

GitHub webhooks degraded causing job queueing

Minor

Resolved after 8 hours 49 minutes, affecting EU Central (eu-central ARM) and EU Central (eu-central x86) and 1 more, on May 19, 2026.

monitoring May 19, 2026, 09:22 PM UTC

We are close to the end of the backlog of the queued tasks and are seeing full recovery in certain runner pools.

Started: May 19, 2026, 02:50 PM UTC
Resolved: May 19, 2026, 11:39 PM UTC
Duration: 8h 49m
Detected by Pingoru: May 19, 2026, 02:50 PM UTC

Affected components

EU Central (eu-central ARM)EU Central (eu-central x86)US West (us-west ARM)US West (us-west x86)EU West (eu-west x86)US Central (us-central MacOS)

Update timeline

investigating May 19, 2026, 02:50 PM UTC

We're seeing evidence of webhooks delivery being degraded from GitHub. We're investigating.
monitoring May 19, 2026, 03:53 PM UTC

We're seeing upstream recovery for GitHub webhook deliveries. Jobs may queue as we process GitHub's backlog of webhook events.
monitoring May 19, 2026, 05:10 PM UTC

We're still seeing a large backlog of queued jobs due to the incident that the system is working through, we're exploring mitigations.
monitoring May 19, 2026, 08:02 PM UTC

We're still working through a substantial backlog of queued tasks that has accumulated over this period of delayed webhook arrivals.
monitoring May 19, 2026, 09:22 PM UTC

We are close to the end of the backlog of the queued tasks and are seeing full recovery in certain runner pools.
resolved May 19, 2026, 11:39 PM UTC

This incident has been resolved.

Your product runs on more than Blacksmith

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.