Grain incident

Failure to start recordings

Resolved after 4 hours 25 minutes, affecting Grain Recorder, on March 24, 2026.

resolved Mar 24, 2026, 07:58 PM UTC

Our upstream provider has closed their incident. We are no longer observing any instability. Cleanup of captured but unprocessed recordings is underway.

Started
Mar 24, 2026, 03:32 PM UTC
Resolved
Mar 24, 2026, 07:58 PM UTC
Duration
4h 25m
Detected by Pingoru
Mar 24, 2026, 03:32 PM UTC

Affected components

Grain Recorder

Update timeline

  1. investigating Mar 24, 2026, 03:32 PM UTC

    An outage with an upstream provider is causing failures for new recordings (both bot and local capture). The team is investigating potential mitigations.

  2. investigating Mar 24, 2026, 03:40 PM UTC

    The team has moved to a new region for the upstream provider and new bot recordings are starting to succeed.

  3. investigating Mar 24, 2026, 05:02 PM UTC

    We are seeing signs that the main region is coming back online. Local captures should start capturing correctly.

  4. monitoring Mar 24, 2026, 05:46 PM UTC

    We've moved back to the main region and we're seeing new bots successfully be launched. Some issues are lingering around double scheduled bots and we are making adjustments to clean those up.

  5. monitoring Mar 24, 2026, 06:31 PM UTC

    The team rolled back to the backup region due to continued higher than normal error rates.

  6. resolved Mar 24, 2026, 07:58 PM UTC

    Our upstream provider has closed their incident. We are no longer observing any instability. Cleanup of captured but unprocessed recordings is underway.

Your product runs on more than Grain

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →