Grain Outage History
Grain is up right nowPingoru has recorded 52 incidents for Grain: 16 in the last 12 months and 4 in the last 90 days.
Incidents, newest first
Minor October 28, 2026 Spike in unprocessed local captures 3h 24m Recording Processing
Started Oct 28, 2026, 05:49 PM UTC· resolved Oct 28, 2026, 09:13 PM UTC
- investigating Oct 28, 2026, 05:49 PM UTC
The team is investigating an increased rate of local captures failing to generate live transcripts. Once the root cause is addressed, the team will reprocess all available captures.
- resolved Oct 28, 2026, 09:13 PM UTC
An upstream transcript provider outage caused live transcription to fail for local capture. We have recovered all impacted captures.
Minor October 21, 2026 Google Meet link format change 1h 59m Grain Recorder, Grain Web App
Started Oct 21, 2026, 03:33 PM UTC· resolved Oct 21, 2026, 05:32 PM UTC
- identified Oct 21, 2026, 03:33 PM UTC
Google Meet is rolling out a new format of meeting links that requires updates to our system. We are rolling out fixes immediately to handle these new formats.
- identified Oct 21, 2026, 04:13 PM UTC
Github outage is causing delays in patches as our release pipeline is reliant on Github workflows.
- resolved Oct 21, 2026, 05:32 PM UTC
Hotfixes have been deployed across all platforms and testing shows capture is working correctly.
Major October 20, 2026 Webhook consumers failing causing app failures 1d 5h Recording Processing, Grain Recorder
Started Oct 20, 2026, 05:38 PM UTC· resolved Oct 21, 2026, 11:03 PM UTC
- investigating Oct 20, 2026, 05:38 PM UTC
The team is actively rolling back a failed update for its webhook consumers. Webhooks are currently failing to process leading to UI to fail to update for bots and recording processing. No data loss has occurred.
- monitoring Oct 20, 2026, 06:05 PM UTC
The team has deployed a rollback and webhooks are correctly process. The team is determining if any recording process needs manual intervention.
- monitoring Oct 20, 2026, 06:33 PM UTC
The team is manually reprocessing all dangling completed recordings as they become available. All new recordings are processing correctly.
- resolved Oct 21, 2026, 11:03 PM UTC
All bot recordings that were left unprocessed due to the outage have been recovered.
Minor September 16, 2026 Transcription Provider Outage 8h 5m Recording Processing
Started Sep 16, 2026, 08:48 PM UTC· resolved Sep 17, 2026, 04:54 AM UTC
- identified Sep 16, 2026, 08:48 PM UTC
Our primary transcription provider is experiencing an outage, the team is monitoring for impact and any required recovery.
- resolved Sep 17, 2026, 04:54 AM UTC
The upstream incident has been resolved, the team will work through the backlog of any impacted recordings for reprocessing.
Notice June 26, 2026 Bot Delays and Failures –
Started Jun 26, 2026, 07:45 PM UTC· resolved Jun 26, 2026, 07:45 PM UTC
- resolved Jun 26, 2026, 07:45 PM UTC
An outage with an upstream provider caused recording bots to be delayed or fail to join altogether. The team is investigating potential mitigations for the future.
Minor June 25, 2026 Transcript Partner Outage 4d 2h Recording Processing
Started Jun 25, 2026, 06:01 PM UTC· resolved Jun 29, 2026, 08:49 PM UTC
- identified Jun 25, 2026, 06:01 PM UTC
Our transcript partner experienced an outage causing some captures to fail to generate a transcript. The team is actively working to reprocess all impacted captures.
- resolved Jun 29, 2026, 08:49 PM UTC
All failed transcripts have been reprocessed.
Notice June 23, 2026 App outage 6h 25m Grain Web App
Started Jun 23, 2026, 04:15 PM UTC· resolved Jun 23, 2026, 10:40 PM UTC
- monitoring Jun 23, 2026, 04:15 PM UTC
The team identified a crashing bug during a deployment, a hot fix was applied and the application is once again loading. The team is monitoring for further issues.
- resolved Jun 23, 2026, 10:40 PM UTC
The team has no observed further issues.
Major June 11, 2026 Stuck local captures 11d 17h Recording Processing
Started Jun 11, 2026, 11:12 PM UTC· resolved Jun 23, 2026, 04:14 PM UTC
- investigating Jun 11, 2026, 11:12 PM UTC
Some local captures are stuck in a processing state. The team is investigating the root cause but believe that all data will be fully recoverable.
- resolved Jun 23, 2026, 04:14 PM UTC
All hung local captures were recovered.
Major June 11, 2026 Bot failures 6h 6m Grain Recorder
Started Jun 11, 2026, 03:18 PM UTC· resolved Jun 11, 2026, 09:24 PM UTC
- investigating Jun 11, 2026, 03:18 PM UTC
An outage with an upstream provider is causing instability for bots including delays in scheduled bots joining and ad-hoc bot failures. The team is investigating potential mitigations.
- monitoring Jun 11, 2026, 03:40 PM UTC
At the half hour, the team observed minimal delays in scheduled bot and no ad-hoc failures. We are in communication with the upstream provider about any further potential issues.
- resolved Jun 11, 2026, 09:24 PM UTC
The team has not observed any further degradation in the bot service.
Major June 2, 2026 Bot Recording Failures 1d 1h Grain Recorder
Started Jun 02, 2026, 03:20 PM UTC· resolved Jun 03, 2026, 04:49 PM UTC
- investigating Jun 02, 2026, 03:20 PM UTC
Upstream provider is experiencing an outage causing out instability in our bot recorders including delayed joined times and recording failures. The team is investigating the scope and any potential mitigations.
- monitoring Jun 02, 2026, 06:29 PM UTC
The outage has subsided. The team has identified a set of recordings currently stuck in processing state that require manual recovery and will be working to recover them as quickly as possible.
- resolved Jun 03, 2026, 04:49 PM UTC
All recordings that were successfully captured have now been recovered. Future improvements around being more robust to upstream failures are being researched by the team.
Minor May 13, 2026 Recording Processing Issues 1d 10h Grain Recorder
Started May 13, 2026, 05:55 PM UTC· resolved May 15, 2026, 04:15 AM UTC
- investigating May 13, 2026, 05:55 PM UTC
An issue with an upstream provider is causing some bot recordings to be stuck processing. We are working with the upstream provider to uncover the root cause an to recover the recordings.
- investigating May 14, 2026, 04:26 PM UTC
Most recordings impacted by the issue have successfully recovered, a small set are still pending further investigation. The recorder has returned to full functionality.
- resolved May 15, 2026, 04:15 AM UTC
All recordings with recoverable media have now be successfully recovered.
Minor May 13, 2026 Platform instabilitiy 6h 8m Grain Desktop App, Grain Web App
Started May 13, 2026, 04:47 PM UTC· resolved May 13, 2026, 10:55 PM UTC
- investigating May 13, 2026, 04:47 PM UTC
Platform is experiencing stability due to increased database load. The team is investigating and looking for mitigations.
- monitoring May 13, 2026, 05:48 PM UTC
The team has deployed a fix and the database load is now back to normal. We are monitoring for any further issues.
- resolved May 13, 2026, 10:55 PM UTC
The team determined the root cause was related to index changes on a hot job queue table, leading to high CPU usage. The correct indexes were applied and the database returned to full health.
Critical March 24, 2026 Failure to start recordings 4h 25m Grain Recorder
Started Mar 24, 2026, 03:32 PM UTC· resolved Mar 24, 2026, 07:58 PM UTC
- investigating Mar 24, 2026, 03:32 PM UTC
An outage with an upstream provider is causing failures for new recordings (both bot and local capture). The team is investigating potential mitigations.
- investigating Mar 24, 2026, 03:40 PM UTC
The team has moved to a new region for the upstream provider and new bot recordings are starting to succeed.
- investigating Mar 24, 2026, 05:02 PM UTC
We are seeing signs that the main region is coming back online. Local captures should start capturing correctly.
- monitoring Mar 24, 2026, 05:46 PM UTC
We've moved back to the main region and we're seeing new bots successfully be launched. Some issues are lingering around double scheduled bots and we are making adjustments to clean those up.
- monitoring Mar 24, 2026, 06:31 PM UTC
The team rolled back to the backup region due to continued higher than normal error rates.
- resolved Mar 24, 2026, 07:58 PM UTC
Our upstream provider has closed their incident. We are no longer observing any instability. Cleanup of captured but unprocessed recordings is underway.
Minor March 23, 2026 Platform Instability 4d 5h Grain Desktop App, Recording Processing, Grain Web App
Started Mar 23, 2026, 03:08 PM UTC· resolved Mar 27, 2026, 09:02 PM UTC
- investigating Mar 23, 2026, 03:08 PM UTC
The platform is currently experiencing instability, the team is deploying upgraded infrastructure and investigating the root cause.
- investigating Mar 23, 2026, 03:53 PM UTC
The team has rolled out mitigations and we're not longer seeing node failures. However we are still investigating the root cause of the issue.
- monitoring Mar 24, 2026, 05:08 PM UTC
The mitigation has prevented further instability but the team is still investigating the root cause of the issue. The issue caused intermittent and cascading excess memory pressure on the application nodes.
- resolved Mar 27, 2026, 09:02 PM UTC
The team has deployed a fix around large memory spikes trigging out-of-memory failures. Since the latest fix, no further memory spikes have occurred.
Minor November 11, 2025 Meet Recorder Partial Outage 6h 46m Grain Recorder
Started Nov 11, 2025, 05:30 PM UTC· resolved Nov 12, 2025, 12:16 AM UTC
- investigating Nov 11, 2025, 05:30 PM UTC
An issue with an upstream vendor is causing a partial outage of Meet recorders, investigation is underway.
- monitoring Nov 11, 2025, 06:47 PM UTC
A mitigation has been deployed and we are monitoring for for further failures.
- resolved Nov 12, 2025, 12:16 AM UTC
We have not seen subsequent failures.
Minor November 6, 2025 Incident with upstream provider causing some recorders to fail 15h 36m Grain Recorder
Started Nov 06, 2025, 06:45 PM UTC· resolved Nov 07, 2025, 10:21 AM UTC
- investigating Nov 06, 2025, 06:45 PM UTC
We are monitoring an incident with an upstream provider that is causing some meeting recorders to fail to join calls. We are expecting a resolution in the near term.
- resolved Nov 07, 2025, 10:21 AM UTC
This incident has been resolved.
Major October 20, 2025 Outages with upstream providers causing instability 22h 21m Recording Processing, Grain Recorder
Started Oct 20, 2025, 01:09 PM UTC· resolved Oct 21, 2025, 11:30 AM UTC
- monitoring Oct 20, 2025, 01:09 PM UTC
AWS has deployed mitigations but we are still observing some issues with upstream providers being able to launch instances. We are continuing to monitor the situation.
- monitoring Oct 20, 2025, 01:59 PM UTC
We are continuing to monitor for any further issues.
- monitoring Oct 20, 2025, 02:55 PM UTC
AWS continues to be have failures launching new instances which is preventing scheduled and ad-hoc recorders from booting up. The team is working on spinning up resources in a new region.
- monitoring Oct 20, 2025, 04:07 PM UTC
We've deployed a patch to start scheduling the recording in a new AWS Region, we're seeing successful recording runs.
- monitoring Oct 20, 2025, 06:03 PM UTC
The patches for running the recorder in a new region had some unexpected side effects causing some recorders to continue to fail. Further mitigations have been deployed and processing appears to be moving forward successfully.
- resolved Oct 21, 2025, 11:30 AM UTC
This incident has been resolved.
Minor September 9, 2025 Recording processing delay 1d Recording Processing
Started Sep 09, 2025, 04:56 PM UTC· resolved Sep 10, 2025, 05:31 PM UTC
- monitoring Sep 09, 2025, 04:56 PM UTC
Due to an unexpected change in a sub processors API, recording processing experienced a temporary delay. This change has been reverted and we are seeing processing move forward. We are monitoring to ensure all backed up processing completes as expected.
- resolved Sep 10, 2025, 05:31 PM UTC
All delayed recordings have successfully processed.
Minor August 27, 2025 Recording Processing instability 9d 7h Recording Processing, Grain Recorder
Started Aug 27, 2025, 05:41 PM UTC· resolved Sep 06, 2025, 12:51 AM UTC
- investigating Aug 27, 2025, 05:41 PM UTC
Recording capture is currently experiencing issues because of instability with the Grain database. The team is investigating the root cause and deploying mitigations.
- monitoring Aug 27, 2025, 07:48 PM UTC
The system has stabilized and the team will now be working on recovering failed recordings where possible.
- resolved Sep 06, 2025, 12:51 AM UTC
This incident has been resolved.
Critical July 28, 2025 Bots failing to join calls 3h 26m Grain Recorder
Started Jul 28, 2025, 01:21 PM UTC· resolved Jul 28, 2025, 04:47 PM UTC
- investigating Jul 28, 2025, 01:21 PM UTC
We are currently investigating an issue that is causing bots to be unable to join meetings.
- identified Jul 28, 2025, 01:25 PM UTC
Our affected upstream provider has identified the issue and is in the process of deploying mitigations.
- monitoring Jul 28, 2025, 02:11 PM UTC
Our affected upstream provider has deployed mitigations and we are now observing bots joining meetings at normal rates. We will continue to monitor for stability.
- resolved Jul 28, 2025, 04:47 PM UTC
This incident has been resolved.
Minor June 12, 2025 Outages with Upstream Providers causing instability 2h 4m Recording Processing
Started Jun 12, 2025, 07:35 PM UTC· resolved Jun 12, 2025, 09:39 PM UTC
- monitoring Jun 12, 2025, 07:35 PM UTC
Upstream providers including Google services and vendors relying on Google Cloud Platform are experiencing widespread outages which is causing instability and processing delays for Grain. Once the upstream issues are resolved, recording processing should fully complete automatically.
- resolved Jun 12, 2025, 09:39 PM UTC
The upstream outage has resolved and Grain systems are back to full operation.
Notice May 18, 2025 MS Calendar Sync Outage 1h
Started May 18, 2025, 08:30 PM UTC· resolved May 18, 2025, 09:30 PM UTC
- resolved May 29, 2025, 04:29 PM UTC
From 05/18 ~21:30 UTC until 05/19 ~10:00 UTC, Microsoft calendars were failing to sync due to an expired Microsoft OAuth Client credential. Due to the nature of the failure, alarms were not correctly setup which lead to the delayed resolution. The sync failure led to some events not having their auto-record rules applied if the event was created or edited during the outage window. The team has put in place mitigations to prevent this failure state in the future.
Major March 19, 2025 High Rate of Recorder Failure 70d 23h Grain Recorder
Started Mar 19, 2025, 05:22 PM UTC· resolved May 29, 2025, 04:24 PM UTC
- investigating Mar 19, 2025, 05:22 PM UTC
Our recording service is experiencing a high rate of failures, causing issues with scheduling and booting up new recorders. Some meetings may not be captured.
- monitoring Mar 19, 2025, 06:13 PM UTC
Mitigation efforts have brought the recording service to a stable place, we are monitoring for further issues and will be evaluating any possibilities for recovery of incorrectly handled recordings.
- monitoring Mar 19, 2025, 07:14 PM UTC
We've seen another uptick in errors after the top of the hour and new mitigations are being explored.
- resolved May 29, 2025, 04:24 PM UTC
This incident has been resolved.
Minor February 7, 2025 Application Instability 39d 22h Grain Desktop App, Recording Processing, Grain Web App
Started Feb 07, 2025, 06:44 PM UTC· resolved Mar 19, 2025, 05:21 PM UTC
- identified Feb 07, 2025, 06:44 PM UTC
The application has been experiencing instability due to increased database load caused by a dependency update. This is causing some functions to fail to process including calendar syncing. We're working on rolling back the dependency to resolve the issue.
- monitoring Feb 07, 2025, 07:27 PM UTC
The fix has been deployed and database load has returned to normal. We're monitoring for further issues and evaluating processes that need to be re-run.
- resolved Mar 19, 2025, 05:21 PM UTC
This incident has been resolved.
Minor January 3, 2025 Transcript Processing Delays 30d 22h Recording Processing
Started Jan 03, 2025, 05:33 PM UTC· resolved Feb 03, 2025, 04:26 PM UTC
- investigating Jan 03, 2025, 05:33 PM UTC
Our transcription partner is experiencing an outage, causing delays in transcript processing. We are awaiting further information.
- monitoring Jan 03, 2025, 05:55 PM UTC
The outage is beginning to clear and we are seeing transcripts complete processing. We will begin backfilling any failed transcripts once we've received confirmation that the outage is fully resolved.
- resolved Feb 03, 2025, 04:26 PM UTC
This incident has been resolved.