Currents Outage History

Started Oct 01, 2026, 05:59 PM UTC· resolved Oct 01, 2026, 07:18 PM UTC

1. investigating Oct 01, 2026, 05:59 PM UTC

   We are observing a delay in delivering notifications to 3rd parties and updating analytics events. .

2. identified Oct 01, 2026, 06:01 PM UTC

   The issue is related to an outage in upstream service outage. We are monitoring the systems and working on a workaround. .

3. identified Oct 01, 2026, 06:34 PM UTC

   Implemented a workaround to unblock and process notifications to 3rd party integrations.

4. monitoring Oct 01, 2026, 06:53 PM UTC

   We implemented a workaround and currently are processing the backlog of events and data sync requests..

5. resolved Oct 01, 2026, 07:18 PM UTC

   This incident has been resolved, our workarounds are keeping everything in sync. We are monitoring the upstream dependency outage..

Read the full incident report →

Started May 04, 2026, 12:30 PM UTC· resolved May 04, 2026, 02:49 PM UTC

1. investigating May 04, 2026, 12:30 PM UTC

   We are currently investigating this incident.

2. monitoring May 04, 2026, 02:04 PM UTC

   We implemented a fix and are currently monitoring the result.

3. resolved May 04, 2026, 02:49 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 14, 2026, 09:30 AM UTC· resolved Apr 14, 2026, 03:32 PM UTC

1. investigating Apr 14, 2026, 09:30 AM UTC

   We are currently investigating this incident..

2. identified Apr 14, 2026, 03:07 PM UTC

   The issue was caused by a stalled bullmq queue and a slowness in redis cluster. We have restored the operational capacity, investigating the root cause. .

3. monitoring Apr 14, 2026, 03:16 PM UTC

   We are monitoring ingestion pipeline performance and stability before declaring that the incident is resolved..

4. resolved Apr 14, 2026, 03:32 PM UTC

   This incident has been resolved. A detailed post-mortem will follow. .

5. monitoring Apr 15, 2026, 03:47 AM UTC

   ## Incident Summary On Apr 14 we had a production degradation caused by the shared ops queue becoming saturated. This was not driven by an unusual traffic spike alone. The deeper issue was that a recent change introduced high-volume step-upload work into the hot path of the shared ops queue, so traffic patterns that had been tolerable before now created queue backlog, worker saturation, and customer-visible latency. ## Timeline * 09:32 UTC: jobs started accumulating in the queue and pending duration increased sharply. * 09:32 UTC onward: writer workers saturated and latency-sensitive ops work started competing with a large volume of step-upload jobs. * 11:30 UTC: main Redis cluster CPU reached 100%. * EU team member declared incident and escalated the issue to NA team. The response was significantly delayed due to lack of adequate notification setting on on-call person mobile device. * 15:00 UTC: autoscaling and cleanup reduced the backlog, but in-flight update request expire after \~2 hours - some customers experienced data loss. ## Root Cause * Data ingestion pipeline generates a distinct task to process step-level data, a new task is created in operations queue for every attempt with steps. * Those tasks are delayed and throttles by a fixed interval, which synchronized large batches into thundering herds instead of smoothing load. * A single task then became the dominant queue workload and starved more latency-sensitive ops tasks, spiking workers CPU to 100% and creating cascading effects on the rest of the system. In short: the root cause was architectural. We moved a high-volume task into the shared hot path, and that queue was no longer reserved for the more time-sensitive work it previously handled. ## Contributing Factors * During the last two weeks we significantly refactored our architecture and moved some components to a long-running ECS jobs in order to reduce the number of connections. * We moved step-processing task into the ops queue. * Writer capacity and autoscaling were not aggressive enough for this new workload shape. * We lacked early alerts on worker saturation. ## Alerts We already have alerts for the following infrastructure components: * Alert on ops queue waiting depth. * Alert on writer worker CPU saturation and sustained concurrency saturation. * Alert on Redis CPU saturation and latency. ## Escalation And Communication The most impactful issues was non-adequate escalation. Despite documented escalation procedures and policies, the on-call person notification setting on personal mobile device were silenced. After the issue was resolved, Currents purchased a dedicated on-call SaaS with a dedicated mobile application that bypasses silenced notification settings. We tested and educated all team members on how to use it. ## Technical Follow-ups * Remove hot-path steps processing task the shared ops queue. * Add the alerts above with explicit thresholds and owners..

Read the full incident report →

Started Feb 11, 2026, 05:11 PM UTC· resolved Feb 11, 2026, 10:22 PM UTC

1. investigating Feb 11, 2026, 05:11 PM UTC

   We are currently investigating this incident. But processing results has gotten significantly behind. We are also seeing an increase of errors in the endpoints that receive client requests from the reports..

2. identified Feb 11, 2026, 05:38 PM UTC

   We are continuing to work on recovery. The endpoint errors have been resolved. But our queues are backed up, so results are taking a while to show in the dashboard. .

3. identified Feb 11, 2026, 07:41 PM UTC

   We are continuing work on full recovery. Initial results processing and notifications are back to normal, but some data queues that populate the dashboard charts and explorers are still delayed..

4. monitoring Feb 11, 2026, 10:15 PM UTC

   We are almost fully recovered, though are still seeing a 10 minute delay on our analytics queues, that power some of the reports in the dashboard. These should finish catching up over the next half hour, with the delay improving steadily. .

5. resolved Feb 11, 2026, 10:22 PM UTC

   This incident has been resolved. All queues are caught up..

Read the full incident report →

Started Dec 15, 2025, 04:42 PM UTC· resolved Dec 15, 2025, 06:00 PM UTC

1. identified Dec 15, 2025, 04:42 PM UTC

   Our background tasks queues have gotten behind and as a result run status in the dashboard and event notifications are currently delayed. This was due to a low auto-scaling cap put in place during our last incident (to limit database connections). This scaling issue has now been resolved but we will expect it to take some time for the system to catch up. And result notifications will likely continue to be delayed for the next hour or two..

2. monitoring Dec 15, 2025, 05:23 PM UTC

   The queue has now recovered, we will monitor the change as the extra processes we launched ramp back down to normal levels..

3. resolved Dec 15, 2025, 06:00 PM UTC

   This incident has been resolved..

Read the full incident report →

Started Dec 13, 2025, 02:30 AM UTC· resolved Dec 13, 2025, 05:20 AM UTC

1. investigating Dec 13, 2025, 02:30 AM UTC

   We are currently investigating this incident..

2. resolved Dec 13, 2025, 05:20 AM UTC

   This incident has been resolved. The preliminary root cause is a sudden spike in concurrent executions, which affected the DB capacity. After increasing the capacity and processing the backlog the system is back to normal. A more thorough investigation and mitigation are to follow to improve system stability due to surge in concurrent requests..

Read the full incident report →

Started Oct 20, 2025, 10:00 AM UTC· resolved Oct 20, 2025, 10:21 PM UTC

1. investigating Oct 20, 2025, 10:00 AM UTC

   Our auto-scaling is currently impacted due to an AWS EC2 outage. We are waiting for new instances to be provisioned in order to resume and complete all pending tasks..

2. identified Oct 20, 2025, 12:44 PM UTC

   Due to AWS us-east-1 outage we are still experiencing issues. There is a big backlog of data update tasks we are unable to process because of lack of EC2 resources. We are looking for alternative runtimes to restore the functionality. Relevant excerpt from AWS: \[04:48 AM PDT\] We continue to work to fully restore new EC2 launches in US-EAST-1\. We recommend EC2 Instance launches that are not targeted to a specific Availability Zone (AZ) so that EC2 has flexibility in selecting the appropriate AZ. The impairment in new EC2 launches also affects services such as RDS, ECS, and Glue. We also recommend that Auto Scaling Groups are configured to use multiple AZs so that Auto Scaling can manage EC2 instance launches automatically..

3. resolved Oct 20, 2025, 10:21 PM UTC

   Our systems are back to normal. After AWS restored their services we were able to start processing the incoming data. There's still a backlog of non-processes event accumulated during the outage. * Our focus is on processing the newly created runs without any delay. * Due to nature of our service there's less value in real-time processing of the delayed events because the associated runs are already expired and timed out. * The backlog of post-process still to be process for analytics and performance analysis. This outage revealed a few performance and resilience related issues with our system. We will follow up with a more detailed analysis..

Read the full incident report →

Started Aug 21, 2025, 06:23 PM UTC· resolved Aug 21, 2025, 08:24 PM UTC

1. identified Aug 21, 2025, 06:23 PM UTC

   Due to a network issue, uploads may be slow. Cypress uploads of logs in all regions are impacted (because our ingest server for these is in the impacted area), and playwright artifact uploads from the us east coast..

2. monitoring Aug 21, 2025, 07:06 PM UTC

   The issue looks to have been resolved. We are continuing to monitor and confirm..

3. resolved Aug 21, 2025, 08:24 PM UTC

   This incident has been resolved..

Read the full incident report →

Started Aug 18, 2025, 03:33 PM UTC· resolved Aug 18, 2025, 06:17 PM UTC

1. investigating Aug 18, 2025, 03:33 PM UTC

   We are currently investigating this incident..

2. monitoring Aug 18, 2025, 04:46 PM UTC

   We are no longer experiencing the issue, but are still investigating the cause. .

3. resolved Aug 18, 2025, 06:17 PM UTC

   Incident resolved. We had a partial outage with our object storage provider that resulting in some uploads and requests timing out. We also had issues with our retry logic triggering access limit. We will look into improving this area..

Read the full incident report →

Started May 15, 2025, 07:05 PM UTC· resolved May 15, 2025, 07:17 PM UTC

1. identified May 15, 2025, 07:05 PM UTC

   A failed deployment caused a 10-minutes outage. During that outage test reporters were not able to upload artifacts. After receiving the alerts we rolled back to failed deployment and the system went back to normal. .

2. resolved May 15, 2025, 07:17 PM UTC

   This incident has been resolved..

Read the full incident report →

Started May 09, 2025, 12:04 AM UTC· resolved May 09, 2025, 01:08 AM UTC

1. investigating May 09, 2025, 12:04 AM UTC

   We are currently investigating this incident, where our elasticsearch cluster has become unhealthy, and is impacting the processing of results..

2. monitoring May 09, 2025, 12:59 AM UTC

   We implemented a mitigation and are currently monitoring the result. As well as continuing to investigate the cause..

3. resolved May 09, 2025, 01:08 AM UTC

   This incident has been resolved..

Read the full incident report →

Started Apr 28, 2025, 05:11 PM UTC· resolved Apr 28, 2025, 07:56 PM UTC

1. investigating Apr 28, 2025, 05:11 PM UTC

   We are currently investigating this incident..

2. identified Apr 28, 2025, 06:46 PM UTC

   The issue is caused by an increased memory pressure on an infrastructure component. We are performing an adjustment to it capacity and taking care of the pending tasks..

3. resolved Apr 28, 2025, 07:56 PM UTC

   The system is back to normal. ### Initial Technical Analysis * A sudden increase in ingress traffic caused one of the OLAP DB to throttle new write and read requests * That caused an accumulation of backlog tasks - an autoscaling kicked off but couldn't cope with the backlog * Increasing the infrastructure capacity resolved the issue. ### Impact The issue affected test results reported between April 25 \~4:30pm-7:30pm GMT * runs could be marked as timed out * test results reporting is missing or delayed for affected runs * CI jobs displaying warnings and connectivity error.

Read the full incident report →

Started Mar 21, 2025, 02:30 PM UTC· resolved Mar 21, 2025, 05:30 PM UTC

1. monitoring Mar 21, 2025, 02:30 PM UTC

   Some incompatible settings from a failed deployment were left in place after a rollback, and our rest api endpoint was unreachable for the past 3hrs. Spot instance reset requests, and calls directly to our api endpoints were impacted. We implemented a fix and are currently monitoring the result..

2. resolved Mar 21, 2025, 05:30 PM UTC

   The error has been resolved, and the proper configuration restored. An internal issue has been logged to review the incident and look for future deployment improvements..

Read the full incident report →

Started Jan 11, 2025, 08:24 PM UTC· resolved Jan 12, 2025, 04:02 AM UTC

1. investigating Jan 11, 2025, 08:24 PM UTC

   We are currently investigating this incident. We are seeing an elevated number of errors, as well as load on our database. Run processing has been significantly delayed. We are investigating..

2. identified Jan 11, 2025, 11:15 PM UTC

   We have identified a bottleneck in DB writes that caused slow processing and accumulation of tasks in processing queues. Scaling up the DB cluster and terminating long-running operations restored the system stability. Investigating the root case for the increase in DB resources consumption..

3. monitoring Jan 12, 2025, 01:17 AM UTC

   The system is now once again working as normal. During the incident you may have failed to create runs, and run data from the incident window may be missing in the dashboard, leading to timeout out runs and missing results. New runs should now be working as expected as of Sat 11:30pm UTC We are still closely monitoring for further errors. And we will be doing a deeper investigation into the root cause on Monday. Several steps were taken to increase our database capacity during the incident, and canceling long running task. As well as a rollback of of some of our most recent deployed services. It's not yet clear whether the rollback or the cancelling of long running tasks resolved the incident. We will be looking into the root cause deeper during regular business hours. .

4. resolved Jan 12, 2025, 04:02 AM UTC

   This incident has been resolved. The root cause was identified as a bug in telemetry configuration for background tasks queue. Adding telemetry data broke tasks deduplication, causing DB overload due to redundant write requests..

Read the full incident report →

Started Dec 19, 2024, 05:01 PM UTC· resolved Dec 19, 2024, 08:02 PM UTC

1. investigating Dec 19, 2024, 05:01 PM UTC

   We are currently investigating this incident..

2. investigating Dec 19, 2024, 06:01 PM UTC

   We have reverted the most recent release, and the error rates have gone down. We are still experiencing some slowdowns due to the task backlog that grew during the incident, but the system is mostly recovered. Runs that were created during the outage will be marked as timed out, and you may see errors related to those runs in your local clients if they are still in progress. New runs should not be affected, and should now work as expected. We are still actively investigating the root cause..

3. identified Dec 19, 2024, 06:23 PM UTC

   We are still seeing issues as a result of the backlog, we are scaling up to address the backlog of tasks. We are continuing to investigate the cause in the meantime..

4. monitoring Dec 19, 2024, 07:05 PM UTC

   Our backlog has recovered, and the service is back to normal. The root cause appears to have been changes in our now reverted deployment. We will continue to investigate the cause and provide more details after our investigation..

5. resolved Dec 19, 2024, 08:02 PM UTC

   The system has recovered. We will continue to investigate the root cause and will update the description of the incident with the details when we have them..

Read the full incident report →