Investigating reports of elevated API failures in Chicago region

Update timeline

1. investigating Oct 19, 2026, 05:02 PM UTC

   We are currently investigating reports of elevated API errors across our services.

2. monitoring Oct 19, 2026, 05:11 PM UTC

   A fix has been implemented as of 17:00 UTC and we are seeing the errors return back to baseline. We are continuing to monitor before marking this resolved.

3. resolved Oct 19, 2026, 05:34 PM UTC

   Between 16:35 and 17:03 UTC, Chicago region experienced about 11% failure rates in API errors, failed agent dispatches, and issues connecting to rooms. Room Service APIs called from other regions targeting to a room hosted in Chicago region were also affected during this period. The last related errors occurred at 17:02:42 UTC, and all affected services are operating normally since. We will share a detailed RCA with more information soon.

4. postmortem Oct 22, 2026, 12:47 AM UTC

   ## Summary On October 19, 2026, between 16:34 and 17:03 UTC and again between 17:47 and 17:54 UTC, some room management, agent dispatch and SIP API requests failed for rooms hosted in our US Central \(Chicago\) region. Requests from other regions that targeted those rooms also failed. During these windows some rooms could not be created or joined, and some agents were not dispatched. The cause was CPU exhaustion on the node running one of our message bus instances in US Central. A recently deployed overlay network used more CPU than it requested, and because our monitoring tracked CPU per pod rather than per node, we didn't see the node run out. That message bus instance began dropping messages, and a room-coordination service that depended on it stopped completing requests. We have moved the message bus in US Central onto larger nodes and resized other regions at similar risk. Node-level CPU alerting and automatic recovery for the coordination service are in progress. ## Impact Between 16:34 and 17:03 UTC, and again from 17:47 to 17:54 UTC, room management \(RoomService\), agent dispatch \(AgentDispatchService\) and SIP APIs failed for some rooms hosted in US Central. Because every region relies on the same coordination step, requests from other US regions that targeted those rooms failed as well. Affected requests returned a 503. Agent dispatch was the most affected. During the first window, 26.5% of rooms that requested an agent did not receive one. During the second window, 17.5% did not. Rooms hosted in other regions, and requests that did not target US Central rooms, were not affected. ## Root Cause Changes to room state, such as creating a room or dispatching an agent, must be confirmed by several servers before they take effect. A coordination service gathers those confirmations over our internal messaging system. If a quorum doesn't arrive within one second, the change fails. We recently rolled out a new overlay network for traffic between regions. It is more resilient to packet loss, but uses more CPU than its predecessor. In US Central, the message bus had little CPU headroom, and traffic was uneven: one of the three instances carried more than the other two combined, and its load had nearly doubled over the preceding hours. The overlay runs on every node and carries each message the message bus sends or receives. To guard against packet loss, it sends each message over several relay paths, so its CPU use climbs with message traffic. So that time-sensitive packet delivery is never throttled, the overlay can use more CPU than it requests, and it runs at a higher scheduling priority than the message bus. On the busiest instance's node, the two together pushed CPU above 90% and starved the message bus. At 16:34 UTC, routine node maintenance moved another messaging server to a new host, and its clients reconnected to the remaining servers. Shortly after, one coordination instance stopped getting replies through the overloaded server. New requests still reached it, but the responses it needed to finish them did not. Within four minutes it had stopped completing requests entirely. The coordination service is built to tolerate the loss of an instance: when one fails, another takes over its work. But this instance had not failed outright. Some messages still reached it, so it kept reporting itself healthy, and callers kept waiting on it instead of moving on. They retried, and request volume to it rose more than tenfold. Its unfinished updates also blocked changes to the same rooms in other US regions. We ended the first window by replacing the instance manually. The second window began when the instance that took over its busiest work stalled the same way, because the message bus was still short of CPU. Our monitoring did not catch the saturation. We monitored CPU for each pod against its own request, and relied on Kubernetes to keep nodes from being overloaded. That works only while every workload stays within its request. The message bus never exceeded its request, and the overlay's extra use did not show up in pod-level monitoring, so the node's saturation went unseen. Our error-rate alerts fired within four minutes, but nothing pointed to the node. ## Timeline \(UTC\) * 16:34 - Routine node maintenance moves a messaging server in US Central to a new host. API failures begin for rooms hosted in the region. * 16:38 - Automated monitoring detects elevated error rates across several US regions, and our team begins investigating. * 16:55 - The affected room-coordination instance is identified and replaced. * 17:03 - Error rates return to baseline. * 17:35 - We mark the incident resolved on this status page. The root cause is not yet known. * 17:47 - Errors return as the instance that took over the busiest work degrades in the same way. * 17:52 - That instance is replaced. Error rates return to baseline by 17:54. * 17:59 - A further instance showing early signs of the same problem is replaced. * 18:14 - New rooms are placed in other regions while the investigation continues. * 20:53 - CPU exhaustion on the messaging host is identified as the root cause. * 21:36 - Messaging servers in US Central are moved to larger hosts. * 22:51 - US Central resumes hosting new rooms. ## Corrective Actions & Prevention **Done** * **Increased CPU for the message bus in US Central.** The region's message bus now runs on larger nodes. * **Other regions resized.** We checked peak utilization across every region and moved messaging servers in regions at similar risk onto larger hosts. * **Automatic diagnostic capture.** The room-coordination service now records detailed diagnostics if a stall recurs. **Underway** * **Node-level CPU alerting.** We are adding CPU alerts for each node that carries internal traffic, alongside our pod-level monitoring, and alerts on CPU scheduling delays. * **Deadline-miss alerting.** A room-coordination instance that keeps missing its deadline will page our team. * **Automatic recovery for room coordination.** An instance that stops completing requests will hand off its work, and callers will route away from unhealthy instances. * **Retry limits.** Room-coordination requests will use backoff and a retry budget, so a stalled instance can no longer multiply request volume across regions. * **Messaging client reconnects.** Clients will reconnect automatically when a subscription stops receiving messages. * **Reserved CPU for the network overlay.** Hosts that run our messaging servers will reserve CPU for the overlay, so it cannot starve the servers it carries. We sincerely apologize for the disruption. If you have questions or were affected in ways not described here, please reach out to our support team.