Resolved after 22 hours 21 minutes, affecting Recording Processing and Grain Recorder, on October 20, 2025.
The patches for running the recorder in a new region had some unexpected side effects causing some recorders to continue to fail. Further mitigations have been deployed and processing appears to be moving forward successfully.
Affected components
Update timeline
- monitoring Oct 20, 2025, 01:09 PM UTC
AWS has deployed mitigations but we are still observing some issues with upstream providers being able to launch instances. We are continuing to monitor the situation.
- monitoring Oct 20, 2025, 01:59 PM UTC
We are continuing to monitor for any further issues.
- monitoring Oct 20, 2025, 02:55 PM UTC
AWS continues to be have failures launching new instances which is preventing scheduled and ad-hoc recorders from booting up. The team is working on spinning up resources in a new region.
- monitoring Oct 20, 2025, 04:07 PM UTC
We've deployed a patch to start scheduling the recording in a new AWS Region, we're seeing successful recording runs.
- monitoring Oct 20, 2025, 06:03 PM UTC
The patches for running the recorder in a new region had some unexpected side effects causing some recorders to continue to fail. Further mitigations have been deployed and processing appears to be moving forward successfully.
- resolved Oct 21, 2025, 11:30 AM UTC
This incident has been resolved.
Your product runs on more than Grain
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services