Blacksmith incident

Job adoption delays

Major

Resolved after 1 hour 3 minutes, affecting EU Central (eu-central ARM) and EU Central (eu-central x86) and 1 more, on June 17, 2026.

monitoring Jun 17, 2026, 01:48 PM UTC

We implemented a fix and are currently monitoring the result but seeing recovery. Any existing queued jobs wont self recover so please be sure to cancel and rerun those jobs.

Started: Jun 17, 2026, 12:45 PM UTC
Resolved: Jun 17, 2026, 01:48 PM UTC
Duration: 1h 3m
Detected by Pingoru: Jun 17, 2026, 12:45 PM UTC

Affected components

EU Central (eu-central ARM)EU Central (eu-central x86)US West (us-west ARM)US West (us-west x86)EU West (eu-west x86)US Central (us-central MacOS)Github → ActionsGithub → Webhooks

Update timeline

investigating Jun 17, 2026, 12:45 PM UTC

We are receiving reports of job adoption delays, we are currently investigating.
investigating Jun 17, 2026, 01:01 PM UTC

We believe this is related to an issue with Github Webhooks and are still investigating.
identified Jun 17, 2026, 01:14 PM UTC

We have identified the issue as GitHub sending us a high cardinality of malformed webhooks missing critical pieces of information in the payloads. We are working on a patch to work around this as we wait for GitHub to fix the upstream issue.
monitoring Jun 17, 2026, 01:27 PM UTC

Our engineers are implementing a mitigation which should take 20 minutes to deploy. Thank you for your patience.
monitoring Jun 17, 2026, 01:48 PM UTC

We implemented a fix and are currently monitoring the result but seeing recovery. Any existing queued jobs wont self recover so please be sure to cancel and rerun those jobs.
resolved Jun 17, 2026, 01:48 PM UTC

This incident has been resolved.

Your product runs on more than Blacksmith

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.