SimpleVoIP Outage History

Started Oct 26, 2026, 03:57 PM UTC· resolved Oct 26, 2026, 10:18 PM UTC

1. investigating Oct 26, 2026, 03:57 PM UTC

   We are investigating reports of issues with our Administration portal located at http://portal.simplevoip.us. We will provide updates as soon as we have identified the cause and impact of this event.

2. investigating Oct 26, 2026, 04:07 PM UTC

   We are continuing to investigate this issue.

3. investigating Oct 26, 2026, 04:10 PM UTC

   We are currently investigating reported issues affecting Call Center functionality. We have implemented a routing change as a potential mitigation and are monitoring its effectiveness while continuing to investigate the underlying cause. We will provide further updates as more information becomes available.

4. monitoring Oct 26, 2026, 04:19 PM UTC

   We have taken multiple steps to mitigate the reported Call Center issues and are actively monitoring system performance. Our team continues to investigate and address any remaining issues while working to ensure sustained stability. We will provide further updates as we continue to monitor.

5. identified Oct 26, 2026, 04:42 PM UTC

   Call Center service has been largely restored following our recent routing changes. We are continuing to investigate reports of intermittent instability and are actively monitoring service performance. We will provide additional updates as our investigation progresses.

6. identified Oct 26, 2026, 04:56 PM UTC

   We are currently working to apply a fix for the ongoing Call Center instability. This process involves rolling server restarts, which are not expected to impact phone service or call routing. We continue to monitor service performance closely and will provide further updates as we progress.

7. monitoring Oct 26, 2026, 05:24 PM UTC

   The patch has been successfully applied, and we have moved into monitoring. We are continuing to keep a close eye on Call Center service to ensure stability. Customers who continue to experience issues with Call Center functionality are encouraged to contact our support team.

8. resolved Oct 26, 2026, 10:18 PM UTC

   Following the successful application of our patch and subsequent service restarts, Call Center services have remained stable with no further issues observed. We will continue routine monitoring and are marking this incident as resolved.

Read the full incident report →

Started Oct 21, 2026, 11:00 PM UTC· resolved Oct 21, 2026, 11:00 PM UTC

1. resolved Oct 21, 2026, 11:00 PM UTC

   Yealink devices running older firmware versions (<v84) that fetched their configurations from affected web servers (roughly 10% between 10/16-10/20, 100% on 10/21) had their configurations wiped and could not reconnect to SimpleVoIP without manual intervention. The vast majority of Yealink devices are running newer firmware versions and were not affected.

2. postmortem Oct 23, 2026, 10:14 PM UTC

   ## Impact Yealink devices running older firmware versions \(<v84\) that fetched their configurations from affected web servers \(roughly 10% of check-ins between 10/16-10/20, 100% on 10/21\) had their configurations wiped and could not reconnect to SimpleVoIP without manual intervention. The vast majority of Yealink devices are running newer firmware versions and were not affected. ## Root Cause SimpleVoIP recently replaced some of our web servers in order to keep up to date on bug and security fixes. As part of that server replacement, we introduced an unintended behavior change that redirected unrecognized URLs to our index page, whereas they would previously have received a 404. Yealink phones request some generic boot files \(e.g. /y000000000100.cfg\) before requesting their MAC-specific configurations. We supply everything via the MAC-specific config, so these generic requests are ignored. The old servers responded with a 404. After moving traffic to servers running the new OS/configuration, those requests returned HTML rather than a 404 or valid config documents. Yealink firmware versions newer than v84 ignored these responses as invalid. Older firmware versions attempted to apply the HTML response as a config, found no valid config directives within the response, and blanked the running config, including the provisioning server directive, which prevented the phones from requesting a new config from SimpleVoIP after a reboot. Factory resetting the phone would typically hit the Yealink redirect server, providing it a skeleton config with SimpleVoIP’s provisioning server URL to enable it to download its full config from us. Unfortunately, Yealink changed their redirect server infrastructure on October 1, 2025, making it inaccessible to older firmware versions and many EOL models. As a result, the same models impacted by the invalid generic config responses were unable to reconnect to SimpleVoIP without more significant effort \(providing a provisioning server URL via DHCP option 66 or manually updating phone configs individually\). ## Remediation ### Immediate Our infrastructure team updated the running web server configs to respond with 404s to the generic config requests from these phones at around 6pm CDT Wednesday. This allowed the phones to boot normally when pointed to our provisioning server. ### Near- to Mid-term We will be updating firmware for these older devices to the latest firmware versions available. Some, like the SIP-T29G or SIP-T42G, never received a firmware release from Yealink that would have prevented this issue or allowed them to recover with a one-touch factory reset. Our Operations team will discuss replacement of these models with phones that are still receiving manufacturer updates. It is common to encounter network issues \(usually firewall\) that may prevent phones from pulling the firmware we define in the device config. We’ll be making a concerted effort to notify customers proactively about devices that aren’t updating and working with them to get as close to 100% compliance as possible. Our Engineering team will also be implementing some anomaly tracking within our monitoring system focused on config requests so that we can proactively detect any sudden changes to config retrieval. ## Incident Timeline 2026\.10.16 17:00 UTC - Affected servers were entered into the available server pool serving a small percentage of config traffic. Bulk of traffic still handled by legacy servers. 2026\.10.17-10.20 - Isolated reports of devices not coming up after check-in. 2026\.10.21 05:00 UTC – Newer web servers take all traffic from legacy server pool, increasing percentage of requests handled by affected server. 2026\.10.21 23:00 UTC – Web servers are reconfigured to respond with 404s for generic config requests, allowing successful device config fetches.

Read the full incident report →

Started Oct 16, 2026, 05:00 PM UTC· resolved Oct 21, 2026, 07:45 PM UTC

1. resolved Oct 21, 2026, 11:00 PM UTC

   Affected Algo, Poly VVX 150, 250, 350, 450, and PolyEdge 350 devices that requested their configuration from the affected server. Devices would receive a blank config and fail to connect to SimpleVoIP services until the device pulled a valid configuration from an unaffected server. Poly devices could recover with a reboot, whereas Algo devices needed manual intervention to restore the provisioning server URL.

2. postmortem Oct 23, 2026, 10:10 PM UTC

   ## Impact Affected Algo, Poly VVX 150, 250, 350, 450, and PolyEdge 350 devices that requested their configuration from the affected server. Devices would receive a blank config and fail to connect to SimpleVoIP services until the device pulled a valid configuration from an unaffected server. Poly devices could recover with a reboot, whereas Algo devices needed manual intervention to restore the provisioning server URL. ## Root Cause SimpleVoIP recently replaced some of our web servers in order to keep up to date on bug and security fixes. One of the new servers was missing configuration template files. A software bug in our config generator continued to build a configuration despite not having a template to build from, so Algo and certain Poly phone models requesting their configs from this server received an empty configuration containing just boilerplate. This wiped the SIP credentials and kept the phone from registering until a valid configuration replaced it. ## Remediation ### Immediate Our infrastructure team restored the missing configuration template files to the server in question. Since Poly phones receive their provisioning server setting from a separate configuration file that wasn’t affected by this bug, a simple reboot was enough to tell them to request a new config, at which point they got a completed, valid config file. Algo devices needed to have their provisioning server URL configured either manually through the device UI or via DHCP option 66. ### Near- to Mid-term A patch to fix the partial config bug will be released after appropriate QA to prevent similar issues from reoccurring. We expect that to go live within the next week. Our Engineering team will also be implementing some anomaly tracking within our monitoring system focused on config requests so that we can proactively detect any sudden changes to config retrieval. ## Incident Timeline 2026\.10.16 17:00 UTC - Affected server was entered into the available server pool serving a small percentage of config traffic. Bulk of traffic still handled by legacy servers or healthy replacements. 2026\.10.17-10.20 - Isolated reports of devices not coming up after check-in. Reboots brought devices up in almost all cases. 2026\.10.21 05:00 UTC – Newer web servers take all traffic from legacy server pool, increasing percentage of requests handled by affected server. 2026\.10.21 19:45 UTC – Config templates are added to the affected server, healthy configs are served from this point on.

Read the full incident report →

Started Feb 20, 2026, 07:02 PM UTC· resolved Feb 20, 2026, 08:14 PM UTC

1. investigating Feb 20, 2026, 07:02 PM UTC

   We are investigating reports of some calls not completing on our system. We will provide updates as soon as we have identified the cause and impact of this event.

2. investigating Feb 20, 2026, 07:13 PM UTC

   We have identified that this issue appears to be related to an upstream carrier and is correlated with the ongoing Cloudflare outage affecting portions of internet routing. As a result, any call routes that traverse infrastructure impacted by the Cloudflare issue may experience disruption — even where the core SimpleVoIP platform remains fully operational. At this time: Outbound calling is fully operational. Inbound calling remains impacted for a subset of DIDs, depending on carrier routing paths. We are actively working with our carrier partners and monitoring the broader Cloudflare incident. We will continue to provide updates as more information becomes available.

3. monitoring Feb 20, 2026, 07:36 PM UTC

   Inbound calling has been restored and call traffic has normalized as of approximately 10 minutes ago. Upstream carriers, in coordination with Cloudflare, have identified and implemented workarounds for the underlying routing issues that were impacting certain call paths. At this time: Inbound calling is operational Outbound calling remains fully operational We are continuing to monitor traffic closely to ensure stability. We will provide a final resolution update once the upstream incident has been fully closed.

4. resolved Feb 20, 2026, 08:14 PM UTC

   Call routing has fully stabilized and traffic patterns have returned to normal. The earlier disruption was tied to upstream carrier routing paths correlated with the broader Cloudflare incident. Upstream providers have implemented mitigations and routing has remained stable following those changes. All inbound and outbound calling services are fully operational at this time. We will continue passive monitoring, but no further impact is expected. Thank you for your patience while this was investigated and resolved.

Read the full incident report →

Started Jan 27, 2026, 08:44 PM UTC· resolved Jan 28, 2026, 05:47 AM UTC

1. identified Jan 27, 2026, 08:44 PM UTC

   We are experiencing degraded voice quality and one-way audio issues in the US-Central region due to a CloudFlare issue impacting Chicago. Our engineers have routed traffic to our US-East and US-West datacenters to minimize the impact.

2. monitoring Jan 27, 2026, 09:03 PM UTC

   Most devices have now moved to unaffected zones, so there should be no additional impact at this time. We'll keep an eye on CloudFlare's Status Page (https://www.cloudflarestatus.com/) to determine when it's safe to return the US-Central traffic to its default route.

3. resolved Jan 28, 2026, 05:47 AM UTC

   CloudFlare has resolved the upstream issue, so we have moved normal traffic back to US-Central. Please reach out to our Support team if you are experiencing any service issues.

Read the full incident report →

Started Nov 13, 2025, 08:21 PM UTC· resolved Nov 14, 2025, 04:04 AM UTC

1. identified Nov 13, 2025, 08:21 PM UTC

   We are experiencing some network connectivity issues in our US-West datacenter. Most phones have already re-registered to a healthy zone, and carrier traffic has been moved to avoid any impact. Calls in progress through US-West may have dropped prior to phone re-registration. There should be no further impact once all phone registrations have finished migrating.

2. monitoring Nov 13, 2025, 08:49 PM UTC

   We found a fiber-related issue in US-West and have resolved it. Everything should be stable there now. We're going to maintain the zone override until after hours tonight, then we'll revert traffic routing to normal.

3. resolved Nov 14, 2025, 04:04 AM UTC

   All US-West traffic has been returned to its default route, and everything has been stable since the fiber repair.

Read the full incident report →

Started Oct 20, 2025, 05:40 PM UTC· resolved Oct 21, 2025, 12:22 AM UTC

1. identified Oct 20, 2025, 05:40 PM UTC

   Our Admin Portal (https://admin.simplevoip.us) is failing to load a significant portion of the time due to an underlying AWS network issue. Users may see "Too many redirects" or "Gateway timeout" errors when attempting to log in or navigate within the portal. We are working to mitigate this on our end while AWS continues to work on their issue. The core voice platform should be unaffected, and all call traffic and phone registrations appear normal in our monitoring.

2. resolved Oct 21, 2025, 12:22 AM UTC

   Amazon has declared their incident resolved as of about 90 minutes ago. We saw our web services stabilize for the most part a few hours before.

Read the full incident report →

Started Jul 31, 2025, 11:39 AM UTC· resolved Jul 31, 2025, 03:51 PM UTC

1. investigating Jul 31, 2025, 11:39 AM UTC

   We are investigating reports of unregistered phones on our system. We will provide updates as soon as we have identified the cause and impact of this event.

2. identified Jul 31, 2025, 11:45 AM UTC

   There is an issue with the voipwebaccess.com domain that is impacting connectivity for all phones that register through that domain. We are working with our DNS providers to get service restored ASAP.

3. monitoring Jul 31, 2025, 12:09 PM UTC

   We have resolved the underlying issue with the voipwebaccess.com domain, and phones are starting to re-register. It may take a few more minutes for all phones to come back online.

4. resolved Jul 31, 2025, 03:51 PM UTC

   At this point 90% of devices affected by this morning's DNS issue with voipwebaccess.com have come back online. If you have a device that has not returned to service yet, please give it a reboot. If that doesn't fix it, please contact our Support team at 855-899-8647 or [email protected] for extended troubleshooting.

Read the full incident report →

Started Apr 15, 2025, 01:32 PM UTC· resolved Apr 16, 2025, 04:44 AM UTC

1. investigating Apr 15, 2025, 01:32 PM UTC

   We are investigating reports of some calls not completing on our system. We will provide updates as soon as we have identified the cause and impact of this event.

2. identified Apr 15, 2025, 02:07 PM UTC

   The impacted calls appear to be limited to the US-Central region. We have identified a SIP gateway server as a common element in all of the provided examples, and we are working to remove it from routing to alleviate the issue while we investigate further.

3. identified Apr 15, 2025, 02:56 PM UTC

   Engineers have performed a routing change to correct the issue in US-Central. Calls are completing normally there now. We also identified a server in US-West subject to the same issue, and we are implementing the same solution there. We should have that resolved within a few minutes.

4. monitoring Apr 15, 2025, 03:04 PM UTC

   The change to US-West has been applied, and we are seeing normal call traffic there as well. At this time there should be no further impact. If you experience any issues, please let our support team know.

5. resolved Apr 16, 2025, 04:44 AM UTC

   All services are stable following the routing changes. A post-mortem analysis will be posted here within the next 2 days.

6. postmortem Apr 21, 2025, 08:51 PM UTC

   # Incident Summary SimpleVoIP Support received reports of inbound and outbound call failures starting at around 8am CDT. Our Engineering team investigated the provided example calls and identified one server in US-Central and one in US-West that were routing network traffic inappropriately. After those server routes were corrected just before 10am CDT, most call traffic was restored, but some desktop app users were unable to connect until an additional change was made at around 3pm CDT. # Root Cause At 2am CDT on April 15, we made a routing change to adjust the default route for outbound carrier traffic. Most of our carrier routes have specific overrides to the default, so we only expected even potential impact to toll free traffic. Unfortunately, the config update and subsequent service restart required for this change impacted the physical interfaces used by two of our servers, generating a routing asymmetry and causing traffic from those servers to be ignored by the rest of our platform. This resulted in a roughly 50% failure rate for call completion and some impact for phone/softphone registration across both the US-Central and US-West zones, which may have appeared as high as 100% failure for certain customer locations depending on traffic/registration distribution. # Resolution The asymmetric route was corrected by adjusting the preferred interface of the two impacted servers to match the rest of our server pool, whose members were still handling traffic normally. By forcing these servers to use a particular interface for all of their traffic, we broke connectivity to some desktop apps that relied on the other interface until a second configuration update was made specifically to repair desktop app connectivity. # Preventive Measures The configurations for these servers have been updated and saved to our configuration management system, which should prevent this specific issue from reoccurring. We only performed brief testing following the planned routing change due to the expected lack of impact. This was a clear oversight, and we would have caught this issue overnight if we had run through our exhaustive test suite. Work is planned to automate more of this post-maintenance testing to enable us to run it more consistently, and we will require the full test suite for all configuration changes moving forward, no matter how harmless they seem. We apologize for the impact to your users and customers. We will continue to improve our monitoring and processes to ensure that the impact of any future server maintenance is minimized.

Read the full incident report →

Started Jan 03, 2025, 12:39 PM UTC· resolved Jan 04, 2025, 08:06 PM UTC

1. identified Jan 03, 2025, 12:39 PM UTC

   A systems issue overnight resulted in many phones losing registration to our servers. Most phones will periodically attempt to re-register and come up on their own. A power cycle will accelerate that process if needed. Some phones are still impacted, and we are working to restore them as quickly as possible.

2. identified Jan 03, 2025, 01:52 PM UTC

   Almost all phones have had their configs restored at this point. Roughly 40% of phones are still offline, but will reconnect when they next pull their config from SimpleVoIP, whether via their automated config fetch or a manual power cycle. If any locations are having trouble restoring their phones, please reach out to our support team.

3. identified Jan 03, 2025, 02:54 PM UTC

   We are still working to restore the configs for a few hundred phones to get them up and running again. We expect to have them fully restored within the next hour.

4. monitoring Jan 03, 2025, 03:21 PM UTC

   All phone configs are restored at this time. Any phones still not registered will likely need a reboot to force a config request before they will come back online.

5. resolved Jan 04, 2025, 08:06 PM UTC

   All but a few phones have been back up and registered since yesterday morning. If any phones do not come back up with a manual power cycle, please call into our Support team.

Read the full incident report →

Started Dec 16, 2024, 07:18 PM UTC· resolved Dec 17, 2024, 04:18 PM UTC

1. investigating Dec 16, 2024, 07:18 PM UTC

   We are investigating reports of some calls not completing on our system. We will provide updates as soon as we have identified the cause and impact of this event.

2. investigating Dec 16, 2024, 07:47 PM UTC

   We are currently investigating an issue where some callers attempting to reach customer DIDs are encountering messages such as "Your call cannot be completed as dialed" or other call failure indicators. At this stage, the issue appears to be isolated to callers from AT&T and Verizon networks. Calls originating from other carriers, such as T-Mobile, are completing without any issues. Our engineering team is actively working to identify the root cause and collaborate with all relevant parties to ensure a prompt resolution. We will continue to share updates as new information becomes available.

3. investigating Dec 16, 2024, 08:18 PM UTC

   We have provided detailed examples of the call failures to the upstream carrier involved. They are engaged and actively investigating the issue.

4. investigating Dec 16, 2024, 08:49 PM UTC

   We have escalated the issue further with the upstream carrier and are actively pushing for updates.

5. investigating Dec 16, 2024, 09:35 PM UTC

   We are continuing to investigate. The scope of the issue remains with Verizon and AT&T failing to connect with some numbers primarily in the central region.

6. investigating Dec 16, 2024, 10:26 PM UTC

   The scope of this issue remains consistent. We are continuing to push on the upstream and failing carriers for updates.

7. investigating Dec 16, 2024, 11:47 PM UTC

   We are continuing to apply pressure on the affected upstream carrier and the failing carriers to expedite a resolution. The overall impact remains minimal, with only a small number of customers experiencing issues with their affected numbers.

8. investigating Dec 17, 2024, 02:34 PM UTC

   We are continuing to push for updates from the affected carrier. The total scope of this issue remains the same, with only a handful of customer numbers affected.

9. resolved Dec 17, 2024, 04:18 PM UTC

   As of approximately 9:00 AM CST, our upstream provider confirmed that the affected carrier implemented adjustments on their end to resolve the issue.

Read the full incident report →

Started Nov 27, 2024, 05:55 PM UTC· resolved Nov 27, 2024, 07:10 PM UTC

1. investigating Nov 27, 2024, 05:55 PM UTC

   We are investigating reports of some calls not completing on our system. We will provide updates as soon as we have identified the cause and impact of this event.

2. monitoring Nov 27, 2024, 06:08 PM UTC

   We observed higher than usual CPU utilization on several servers, which contributed to increased Post-Dial Delay, as well as some outbound call failures. We've restarted the affected servers and are seeing normal CPU utilization.

3. resolved Nov 27, 2024, 07:10 PM UTC

   We've observed that all systems have remained stable since the service restarts. Our Engineering team will continue to monitor closely, though we are marking this as Resolved at this time.

Read the full incident report →

Started Nov 25, 2024, 07:14 PM UTC· resolved Nov 26, 2024, 05:03 AM UTC

1. investigating Nov 25, 2024, 07:14 PM UTC

   We are investigating reports of some calls not completing on our system as well as some slow response times through our Admin Portal. We will provide updates as soon as we have identified the cause and impact of this event.

2. identified Nov 25, 2024, 07:41 PM UTC

   We have identified some elevated server load causing significant post-dial delay on a subset of calls and failures on smaller subset. This also resulted in high response times from our Admin Portal. Our US-Central zone showed the highest incidence rate of these symptoms. As a temporary measure, we have routed some traffic away from US-Central to redistribute the load and alleviate the symptoms while we look for the underlying root cause. Since making that routing change, call traffic and web response times have returned to normal. We will continue to investigate and notify when we have confirmation of a fix from our Engineering team.

3. monitoring Nov 25, 2024, 09:46 PM UTC

   All traffic is back to normal levels and stable following the earlier routing change. Our Engineering team is still working to confirm the root cause.

4. resolved Nov 26, 2024, 05:03 AM UTC

   The temporary routing change has been reverted, and all traffic is stable. An RFO will be made available on this status page once the Engineering team wraps up their investigation and determines next steps.

Read the full incident report →