Balena incident

Elevated Dashboard Errors

Resolved after 1 hour 50 minutes, affecting API and Dashboard, on September 28, 2026.

postmortem Sep 29, 2026, 02:18 PM UTC

On September 28, the balenaCloud dashboard failed to load because some API requests stopped responding. A runtime upgrade to our API interacted badly with a legacy storage library, causing requests that depended on it to hang. We rolled back the upgrade, which restored service. To prevent this from happening again, we are removing the legacy code path involved and adding customizable timeouts for external backends.

Started
Sep 28, 2026, 04:23 PM UTC
Resolved
Sep 28, 2026, 06:14 PM UTC
Duration
1h 50m
Detected by Pingoru
Sep 28, 2026, 04:23 PM UTC

Affected components

APIDashboard

Update timeline

  1. investigating Sep 28, 2026, 04:23 PM UTC

    We're experiencing an elevated level of errors on our Dashboard and are currently looking into the issue.

  2. identified Sep 28, 2026, 04:46 PM UTC

    The issue has been identified and a fix is being implemented.

  3. monitoring Sep 28, 2026, 04:50 PM UTC

    A fix has been implemented and we are monitoring the results.

  4. resolved Sep 28, 2026, 06:14 PM UTC

    This incident has been resolved.

  5. postmortem Sep 29, 2026, 02:18 PM UTC

    On September 28, the balenaCloud dashboard failed to load because some API requests stopped responding. A runtime upgrade to our API interacted badly with a legacy storage library, causing requests that depended on it to hang. We rolled back the upgrade, which restored service. To prevent this from happening again, we are removing the legacy code path involved and adding customizable timeouts for external backends.

Your product runs on more than Balena

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →