VoiceDock Outage History

Started Jul 24, 2026, 12:48 PM UTC· resolved Jul 24, 2026, 01:47 PM UTC

1. investigating Jul 24, 2026, 12:48 PM UTC

   At 14:36 CEST our primary voice node in Nuremberg became unreachable due to a network fault at our datacenter provider (Hetzner, nbg1 region). Inbound and web calls routed through this node were affected. We failed over to our secondary voice node in a separate location and calls have been restored. Any call in progress during the short window may have dropped; no customer data was affected, as account and call data are stored separately and were not impacted. We are now serving from the secondary node and will keep monitoring while the provider resolves the upstream fault. 2026-07-24T12:36:00Z

2. resolved Jul 24, 2026, 01:47 PM UTC

   Post-mortem, 24 July 2026. Voice calls were interrupted for approximately 11 minutes (14:36 to 14:48 CEST). Inbound, outbound and web calls routed through our primary voice node were affected. The dashboard, API, data storage and documentation were not affected, and no customer data was lost or exposed. What happened. Our infrastructure provider, Hetzner, had a network fault in their Nuremberg datacenter that isolated our primary voice node. Our platform automatically detected the failure and began failing over to a secondary node in a separate location. One step in that automated failover, moving our public network address to the secondary node, did not complete on its own: it relies on the provider's management API, which was itself degraded by the same fault. Our team completed that step manually, which restored service. The provider resolved the underlying fault shortly afterwards. What we have changed. We have already rewritten that failover step so it completes reliably on its own, without manual intervention, even when the provider's systems are degraded, and we validated the change the same day with a live failover. We are additionally building a second, independent failover path that does not rely on the provider's systems at all. We hold ourselves to a high reliability standard, and an interruption of this length is more than we accept. We are sorry for the disruption.

Read the full incident report →

Started Jun 29, 2026, 10:51 AM UTC· resolved Jun 29, 2026, 11:06 AM UTC

1. investigating Jun 29, 2026, 10:51 AM UTC

   We're investigating an outage affecting our API and database. Inbound and outbound calls, the API, and dashboard data may be unavailable or degraded. We'll post updates here as we learn more.

2. investigating Jun 29, 2026, 10:51 AM UTC

   Identified: a fault in our underlying infrastructure has isolated our database — the same root cause as our previous outage. The redundancy/failover that prevents this is in active development but not yet live. We're actively monitoring recovery and will update here as soon as service is restored. We apologize for the disruption.

3. investigating Jun 29, 2026, 11:02 AM UTC

   Resolved: service has been fully restored. The underlying infrastructure fault recovered around 11:00 UTC and our database, API and calls are operating normally again. We verified the database recovered cleanly. Total impact was approximately 10:25–11:00 UTC. We recognize this is the second occurrence of this failure mode, and the redundancy/failover work that prevents it is being prioritized. Thank you for your patience.

4. resolved Jun 29, 2026, 11:06 AM UTC

   Post-mortem — API & database outage, 29 June 2026 Summary. Between approximately 10:25 and 11:00 UTC (~35 minutes), our API, database and call handling were unavailable. A network fault in our underlying hosting infrastructure isolated the host that runs our primary database. Service has been fully restored and the database recovered cleanly, with no data loss. Timeline (UTC). - 10:25 — Monitoring detected the database and API becoming unreachable. - 10:27–10:40 — Confirmed the primary database host was isolated at the network layer: the server itself stayed healthy but was cut off from the network (the hosting provider's own management console could not reach it either). - ~10:57 — The host was recovered by our infrastructure provider. - 11:00 — Database, API and calls fully restored; we verified the database recovered cleanly. Root cause. A network switch fault in our hosting infrastructure isolated the single host that runs our primary database — the machine kept running, but was cut off from the network and the internet. The same class of fault has affected us once before. Because our primary database currently runs on a single host, recovery depended on the provider restoring connectivity rather than on us failing over to a standby. Impact. API requests failed, dashboard data was unavailable, and inbound and outbound calls could not be established (call configuration is resolved from the database). No data was lost. What we're doing about it. The redundancy and failover work that lets us switch to a standby database in a separate location — independent of any single host or network segment — is in active development and being prioritized. Database backups are already replicated off-site. We'll keep posting updates as these safeguards go live. We're sorry for the disruption. If this outage affected your integration and you'd like specifics, reach us at [email protected].

5. resolved Jun 29, 2026, 11:06 AM UTC

   **Post-mortem — API & database outage, 29 June 2026** **Summary.** Between approximately 10:25 and 11:00 UTC (~35 minutes), our API, database and call handling were unavailable. A network fault in our underlying hosting infrastructure isolated the host that runs our primary database. Service has been fully restored and the database recovered cleanly, with no data loss. **Timeline (UTC).** - 10:25 — Monitoring detected the database and API becoming unreachable. - 10:27–10:40 — Confirmed the primary database host was isolated at the network layer: the server itself stayed healthy but was cut off from the network (the hosting provider's own management console could not reach it either). - ~10:57 — The host was recovered by our infrastructure provider. - 11:00 — Database, API and calls fully restored; we verified the database recovered cleanly. **Root cause.** A network switch fault in our hosting infrastructure isolated the single host that runs our primary database — the machine kept running, but was cut off from the network and the internet. The same class of fault has affected us once before. Because our primary database currently runs on a single host, recovery depended on the provider restoring connectivity rather than on us failing over to a standby. **Impact.** API requests failed, dashboard data was unavailable, and inbound and outbound calls could not be established (call configuration is resolved from the database). No data was lost. **What we're doing about it.** The redundancy and failover work that lets us switch to a standby database in a separate location — independent of any single host or network segment — is in active development and being prioritized. Database backups are already replicated off-site. We'll keep posting updates as these safeguards go live. We're sorry for the disruption. If this outage affected your integration and you'd like specifics, reach us at [email protected].

Read the full incident report →

Started Jun 17, 2026, 10:57 PM UTC· resolved Jun 18, 2026, 12:35 AM UTC

1. investigating Jun 17, 2026, 10:57 PM UTC

   We are investigating a renewed loss of connectivity to our database and API, affecting voice calls and the dashboard. This appears to be a recurrence of the earlier upstream network incident at our cloud infrastructure provider. We are working to restore service as quickly as possible.

2. investigating Jun 17, 2026, 11:25 PM UTC

   The cause has been identified as a network fault at our upstream cloud infrastructure provider, the same fault that affected us earlier today. The provider's technicians are working on restoring the affected network segment. Our database node itself remained healthy but was unreachable at the network level, and no data has been lost.

3. resolved Jun 18, 2026, 12:35 AM UTC

   Resolved. Our upstream provider restarted the affected systems and connectivity was restored at 00:31 UTC. All services have been operational since. Post-mortem: between 22:55 and 00:31 UTC a recurring network fault at our cloud infrastructure provider isolated the database node that powers VoiceDock, making the API, dashboard and voice calls unavailable for roughly 1 hour and 36 minutes. This was the second occurrence of the same upstream fault within 24 hours. The node itself stayed healthy throughout and no data was lost; the outage was a loss of network connectivity, resolved once the provider restored the affected segment. A single dependency should not be able to take the service down. We are accelerating two improvements already underway: relocating our backups to storage independent of the affected infrastructure, and removing the single point of failure in our data layer through geographic redundancy. We apologise for the disruption and are monitoring closely.

Read the full incident report →

Started Jun 17, 2026, 11:48 AM UTC· resolved Jun 17, 2026, 12:43 PM UTC

1. investigating Jun 17, 2026, 11:48 AM UTC

   We have identified the cause of the current disruption. A network fault at our cloud infrastructure provider's datacenter has interrupted connectivity to part of the VoiceDock platform. This is an upstream incident outside our direct control; the provider's engineers are actively working on a resolution. During this period, inbound and outbound calls and the dashboard may be unavailable. No data has been lost or is at risk — the disruption is limited to network connectivity. We are monitoring closely and will post a further update as soon as service is restored or we have a clearer timeline. We apologise for the inconvenience.

2. investigating Jun 17, 2026, 12:08 PM UTC

   Connectivity has been fully restored. The upstream network incident at our cloud provider has been resolved, and all VoiceDock services — inbound and outbound calls and the dashboard — are operating normally again. We are continuing to monitor to confirm stability. Thank you for your patience, and we apologise for the disruption.

3. resolved Jun 17, 2026, 12:43 PM UTC

   Post-mortem — service disruption on 17 June 2026 Summary. VoiceDock experienced a ~45-minute disruption caused by a network fault at our cloud infrastructure provider's datacenter. Inbound and outbound calls and the dashboard were unavailable during the window. No data was lost or corrupted. Impact (≈11:18–12:04 UTC). Calls could not be configured and the dashboard could not load data, because both depend on our database host, which was isolated at the network level. The host itself stayed healthy — only its datacenter network path failed. Call recordings (stored separately) were unaffected. Timeline (UTC). - 10:54 — last call processed normally. - 11:18 — automated monitoring detected the API and database becoming unreachable; on-call was alerted within ~1 minute. - ~11:30 — investigation confirmed the host was fully unreachable at the network level; cause traced to the provider. - 11:48 — status page updated with the identified cause. - ~12:03 — the provider restored the network segment and the host returned automatically. - 12:08 — all services verified healthy; incident resolved. Root cause. A network switch ("leaf") fault in our provider's datacenter isolated the server hosting our database and platform at the network level — an upstream provider incident outside our direct control. What went well. Sub-minute detection and alerting, customer communication via this status page during the window, and nightly-validated backups (restore and secret decryption both verified). What we're improving. 1. Off-site backups — moving database backups to independent, off-provider storage so recovery never depends on the affected host. 2. Redundancy — evaluating a standby in a second location and a managed, multi-region database option to remove this single-provider, single-location failure mode. We apologise for the disruption and thank you for your patience.

4. investigating Jun 17, 2026, 12:43 PM UTC

   **Post-mortem — service disruption on 17 June 2026** **Summary.** VoiceDock experienced a ~45-minute disruption caused by a network fault at our cloud infrastructure provider's datacenter. Inbound and outbound calls and the dashboard were unavailable during the window. No data was lost or corrupted. **Impact (≈11:18–12:04 UTC).** Calls could not be configured and the dashboard could not load data, because both depend on our database host, which was isolated at the network level. The host itself stayed healthy — only its datacenter network path failed. Call recordings (stored separately) were unaffected. **Timeline (UTC).** - 10:54 — last call processed normally. - 11:18 — automated monitoring detected the API and database becoming unreachable; on-call was alerted within ~1 minute. - ~11:30 — investigation confirmed the host was fully unreachable at the network level; cause traced to the provider. - 11:48 — status page updated with the identified cause. - ~12:03 — the provider restored the network segment and the host returned automatically. - 12:08 — all services verified healthy; incident resolved. **Root cause.** A network switch ("leaf") fault in our provider's datacenter isolated the server hosting our database and platform at the network level — an upstream provider incident outside our direct control. **What went well.** Sub-minute detection and alerting, customer communication via this status page during the window, and nightly-validated backups (restore and secret decryption both verified). **What we're improving.** 1. **Off-site backups** — moving database backups to independent, off-provider storage so recovery never depends on the affected host. 2. **Redundancy** — evaluating a standby in a second location and a managed, multi-region database option to remove this single-provider, single-location failure mode. We apologise for the disruption and thank you for your patience.

Read the full incident report →