LiveKit Outage History

We have resolved the underlying issue and have not observed any further occurrences of this incident. We will follow up with a detailed postmortem in the original incident.

This incident has been resolved. Between 12:08 and 14:23 UTC, customers were unable to access the Cloud Dashboard and the Analytics Services APIs, caused by a loss of network reachability to the infrastructure serving them. We routed traffic to a healthy endpoint, and access returned to normal by 14:23 UTC. Room APIs, RTC APIs, and all other services were unaffected, and we have no indication of data loss. We will follow up with a postmortem.

A subset of room sessions that ended around 4 September 09:57 UTC were not recorded with an end time. These sessions continue to appear as active in the Cloud dashboard and in session records, and no `room_ended` end time is reflected for them. This was a reporting issue only. Rooms ended normally for participants, live traffic was not affected, and there is no impact on usage or billing. The underlying cause has been identified and fixed. Affected sessions from this window may continue to display as active; they can be safely disregarded.

This incident has been resolved. We will follow up with a detailed postmortem as soon as possible.

This incident has been resolved and dashboard load times returned to normal by 1055 UTC

Between 08:47 and 08:54 UTC, the service that routes LLM, STT and TTS requests through LiveKit Inference in our US East region was unavailable, causing inference requests from agents in that region to fail or time out. Sessions depending on those responses may have degraded or ended prematurely. Replacement capacity came online at 08:54 UTC and request volumes returned fully to normal levels by 09:02 UTC. We have confirmed no further impact since then.

Agent session analytics in the Cloud dashboard have recovered and current data is reporting correctly for all affected projects. Some gaps may remain in historical data from the affected window (August 21–24 UTC); these will be backfilled.

From 09:18 UTC on 21 August, 0.008% of egress recordings processed in our US East region were terminated before completion. Affected recordings were marked EGRESS_FAILED ("egress timed out"). For file outputs, media was not written to the configured destination; for HLS and streaming destinations, media was delivered up to the point of failure. The cause was node scale-down evicting egress workers with active recordings. A fix preventing this has been deployed on 23 August 09:00 UTC.

This incident has been resolved. Between 19:20 and 00:10 UTC, session data and observability ingestion was delayed by 45-60 min. The root cause has been addressed and data ingestion returned to normal by 00:10 UTC. No data was lost during this window.

We have resolved the incident as session load times have returned to baseline.

Error rates on ListParticipants API requests recovered at 18:58 UTC and have remained stable since.

This incident has been resolved. Between 19:45 and 20:00 UTC, US West and US Central experienced intermittent degradation in Room Service and SIP performance. We rerouted traffic and performance has returned to baseline as of 22:00 UTC. We will follow up with a postmortem, which will include further details regarding scale and scope.

Today there was an issue with one of our GPU providers that caused it run at about 50% capacity. Requests above capacity return 503s, which we route to another, fallback provider. This fallback provider was not able to rise to the occasion and also returned 503s. In the very near future we will be onboarding additional providers to avoid these kinds of issues.

This incident has been resolved.

We have fixed the data pipeline delay and all sessions should be loading now in the Cloud Dashboard.

Between approximately 10:30 and 11:55 UTC on July 28, customers with telephony workloads in our London region experienced elevated error rates on SIP APIs. Small number of call transfers were affected. We mitigated by removing the affected infrastructure from service at 11:48 UTC, and error rates returned to normal by 11:55 UTC.

On July 27, our San Jose region experienced a minor networking incident beginning at 16:14 UTC. Users connected to this region may have experienced reconnects and increased session start times. Traffic was routed away from this region at 16:39 UTC and errors have since returned to baseline.