Grain incident

High Rate of Recorder Failure

Resolved after 70 days 23 hours, affecting Grain Recorder, on March 19, 2025.

monitoring Mar 19, 2025, 07:14 PM UTC

We've seen another uptick in errors after the top of the hour and new mitigations are being explored.

Started
Mar 19, 2025, 05:22 PM UTC
Resolved
May 29, 2025, 04:24 PM UTC
Duration
70d 23h
Detected by Pingoru
Mar 19, 2025, 05:22 PM UTC

Affected components

Grain Recorder

Update timeline

  1. investigating Mar 19, 2025, 05:22 PM UTC

    Our recording service is experiencing a high rate of failures, causing issues with scheduling and booting up new recorders. Some meetings may not be captured.

  2. monitoring Mar 19, 2025, 06:13 PM UTC

    Mitigation efforts have brought the recording service to a stable place, we are monitoring for further issues and will be evaluating any possibilities for recovery of incorrectly handled recordings.

  3. monitoring Mar 19, 2025, 07:14 PM UTC

    We've seen another uptick in errors after the top of the hour and new mitigations are being explored.

  4. resolved May 29, 2025, 04:24 PM UTC

    This incident has been resolved.

Your product runs on more than Grain

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →