Is Retell AI down?

We are investigating an issue causing web and outbound calls to fail and inbound calls to experience connection delays. We will provide an update as soon as more information is available.

The issue affecting web, outbound, and inbound calls has been resolved. All calling services are operating normally.

### Summary On September 5, a rare race condition in our custom LLM call path caused call scheduling queues across our fleet to stall. Because the failure also made containers appear underutilized, autoscaling never triggered. Between roughly 10:00 and 16:00 PST the impact grew until most containers were stuck and new calls could not be placed. ### Impact * **Web calls, outbound calls, and inbound calls** were all affected. * Inbound calls were the least affected, since they hold the highest priority in our scheduling queue. * Peak impact occurred around 16:00 PST. ### Root Cause The failure required a specific sequence of events: 1. **Stalled initialization.** If a call ended before its custom LLM websocket was established, the call's initialization never completed. The scheduling queue waited indefinitely for that initialization to finish. 2. **Queue exhaustion.** Each container allows a fixed number of concurrent dequeue operations. Enough stalled initializations consumed all available slots, leaving the container unable to pick up new calls. 3. **Misleading scaling signals.** The call-end path still ran and decremented the container's concurrency counter. As a result, stuck containers reported low concurrency and low resource utilization — so no scale-out was triggered. With every container blocked and autoscaling idle, new calls had nowhere to go. This pattern is only reachable through that exact ordering, which is why it went undetected until it occurred at scale. ### Timeline \(PST\) ### Prevention ### Closing We apologize for the disruption. We are treating this as an opportunity to harden our scheduling and scaling infrastructure, and we will ship the remaining action items within a few days.

Due to issues with an upstream provider, we are seeing connection issues for a portion of inbound calls. This is not widespread.

Service should be back to normal. We are continuing to monitor to ensure stability.

This incident has been resolved.

We identified that certain number's inbound is not connecting, investigating. Outbound is not impacted.

The issue was identified as a datacenter issue from underlying telephony provider, where 2 out of 3 datacenters failed to deliver webhook to Retell backend server.

This has been fixed, we are monitoring.

This incident has been resolved.

Outbound batch calls are failing. We are investigating the issue

We are continuing to investigate this issue.

This incident has been resolved.

The issue has been identified and a fix is being implemented.

This incident has been resolved.