Rudderstack incident

Incorrect Delivery for S3 and S3 Datalake

Resolved after 2 hours 15 minutes, affecting Data Processing Pipeline (US) and Warehouses, on May 28, 2025.

resolved May 29, 2025, 01:40 PM UTC

An issue was identified with the delivery module for S3 & S3_DATALAKE destinations, leading to the incorrect duplication of the prefix in the final file path. This resulted in data being uploaded to an unintended location, making it unavailable for any downstream consumption for the duration of the incident. The flag was enabled only for free tier and starter and growth customers. Follow the below recommended steps to recover the data: Prerequisite: Make sure the AWS CLI is installed and configured locally to access the relevant S3 bucket. 1. Go to the Rudderstack configuration page of your S3/S3 Datalake destinations and copy bucket name and prefix. 2. Copy all the files from the double-prefix path to the single prefix path. Command: aws s3 cp s3://{bucket_name}/{prefix}/{prefix}/ s3://{bucket_name}/{prefix}/ --recursive 3. Delete the original files (optional) Command:aws s3 rm s3://{bucket_name}/{prefix}/{prefix}/ --recursive

Started
May 28, 2025, 05:00 PM UTC
Resolved
May 28, 2025, 07:15 PM UTC
Duration
2h 15m
Detected by Pingoru
May 28, 2025, 05:00 PM UTC

Affected components

Data Processing Pipeline (US)Warehouses

Update timeline

  1. identified May 29, 2025, 01:40 PM UTC

    Issue has been identified with S3 and S3 datalake deliveries. Actively working on the fix.

  2. resolved May 29, 2025, 01:40 PM UTC

    An issue was identified with the delivery module for S3 & S3_DATALAKE destinations, leading to the incorrect duplication of the prefix in the final file path. This resulted in data being uploaded to an unintended location, making it unavailable for any downstream consumption for the duration of the incident. The flag was enabled only for free tier and starter and growth customers. Follow the below recommended steps to recover the data: Prerequisite: Make sure the AWS CLI is installed and configured locally to access the relevant S3 bucket. 1. Go to the Rudderstack configuration page of your S3/S3 Datalake destinations and copy bucket name and prefix. 2. Copy all the files from the double-prefix path to the single prefix path. Command: aws s3 cp s3://{bucket_name}/{prefix}/{prefix}/ s3://{bucket_name}/{prefix}/ --recursive 3. Delete the original files (optional) Command:aws s3 rm s3://{bucket_name}/{prefix}/{prefix}/ --recursive

Your product runs on more than Rudderstack

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →