Bitmovin incident

Encoding failures on AWS

Resolved after 1 hour 8 minutes, on September 4, 2025.

resolved Sep 04, 2025, 04:53 AM UTC

# Root Cause Analysis: Encoding Failures on AWS ### Summary On **September 4, 2025**, between **06:31 AM and 08:10 AM CEST**, encoding jobs running on AWS failed due to an issue accessing our S3 storage. ### Root Cause The incident was caused by an error during a routine S3 key rotation. The old access key was deleted before the new key was in use, which temporarily prevented our encoding service from accessing storage. ### Impact * Only encoding jobs running on **AWS** were affected. * Encodings on other cloud providers and all other Bitmovin services were **not impacted**. ### Resolution The configuration was corrected at **08:10 AM CEST**, restoring access to S3\. Encoding operations on AWS recovered immediately and have been stable since. ### Preventive Measures To prevent this from happening again, we are: * Updating our key rotation procedure to ensure keys are not deleted prematurely. * Automating the key rotation process to reduce the chance of operator error..

Started
Sep 04, 2025, 03:45 AM UTC
Resolved
Sep 04, 2025, 04:53 AM UTC
Duration
1h 8m
Detected by Pingoru
Sep 04, 2025, 03:45 AM UTC

Update timeline

  1. investigating Sep 04, 2025, 03:45 AM UTC

    We are currently experiencing failures with encodings on AWS. Our engineering team is currently investigating the issue.

  2. resolved Sep 04, 2025, 04:53 AM UTC

    # Root Cause Analysis: Encoding Failures on AWS ### Summary On **September 4, 2025**, between **06:31 AM and 08:10 AM CEST**, encoding jobs running on AWS failed due to an issue accessing our S3 storage. ### Root Cause The incident was caused by an error during a routine S3 key rotation. The old access key was deleted before the new key was in use, which temporarily prevented our encoding service from accessing storage. ### Impact * Only encoding jobs running on **AWS** were affected. * Encodings on other cloud providers and all other Bitmovin services were **not impacted**. ### Resolution The configuration was corrected at **08:10 AM CEST**, restoring access to S3\. Encoding operations on AWS recovered immediately and have been stable since. ### Preventive Measures To prevent this from happening again, we are: * Updating our key rotation procedure to ensure keys are not deleted prematurely. * Automating the key rotation process to reduce the chance of operator error..

Your product runs on more than Bitmovin

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →