Viirtue Outage History

Started Jun 27, 2025, 03:37 PM UTC· resolved Jun 30, 2025, 01:25 PM UTC

1. investigating Jun 27, 2025, 03:37 PM UTC

   Viirtue engineers will be performing an emergency maintenance on the Florida Recording server to prevent a possible issue. Some recordings may be unavailable or missing during this short window.

2. investigating Jun 27, 2025, 05:31 PM UTC

   Our engineers are actively working to restore full functionality to the recording server. We’ll share further updates as progress is made.

3. investigating Jun 27, 2025, 06:52 PM UTC

   The issue affecting the recording server has been identified. Our team is now implementing the necessary fixes to restore normal operation.

4. investigating Jun 27, 2025, 08:50 PM UTC

   Our team has made meaningful headway in the investigation and continues to work through targeted areas of the system. We’ll provide further updates as we progress through the next stages.

5. investigating Jun 27, 2025, 10:13 PM UTC

   The recording server issue has been resolved and functionality has been restored. We appreciate your patience while our team worked through the incident.

6. resolved Jun 30, 2025, 01:25 PM UTC

   This incident was resolved on Friday. Correcting statuspage.

Read the full incident report →

Started Jun 20, 2025, 07:19 PM UTC· resolved Jun 20, 2025, 09:51 PM UTC

1. identified Jun 20, 2025, 07:19 PM UTC

   We have identified a Lumen outage in Southern Florida that is causing network connectivity issues in our Florida datacenter. Updates to follow.

2. identified Jun 20, 2025, 07:22 PM UTC

   Due to the minor packet loss in the Florida datacenter, we will be 503'ing the core2-fl server and moving traffic to our Las Vegas core.

3. monitoring Jun 20, 2025, 08:03 PM UTC

   Our monitoring tools indicate that the packet loss has subsided. However, the 503 will remain in effect until we receive confirmation from Lumen that the issue has been fully resolved.

4. monitoring Jun 20, 2025, 09:12 PM UTC

   The network congestion has been reported resolved by lumen, our team will reroute traffic back to this data center shortly. Lumen Ticket: 2025-06-20 21:05:31 GMT - The Lumen NOC has confirmed that backbone links are progressively restoring as fiber repairs are completed. Key transport paths have recovered, alleviating congestion. Services have stabilized as of 20:30 GMT. A final update will follow shortly. 2025-06-20 20:52:52 GMT - Multiple unrelated fiber incidents are currently affecting Florida markets, resulting in significant congestion across several areas. As restoration efforts progress in each location, congestion levels are beginning to ease. The Lumen Network Operations Center (NOC) will continue to monitor the situation closely to ensure stability throughout the repair process.

5. resolved Jun 20, 2025, 09:51 PM UTC

   The 503 has been removed from the Florida core. Service has been fully restored to the Florida datacenter.

Read the full incident report →

Started Jun 18, 2025, 02:30 PM UTC· resolved Jun 18, 2025, 02:30 PM UTC

1. resolved Jun 18, 2025, 03:34 PM UTC

   At approximately 10:13 AM EST on June 18, our core system in New Jersey experienced an unexpected crash. Our platform’s self-healing mechanisms immediately detected the issue and automatically restored service with no sustained downtime. Our engineering team is actively investigating the root cause in collaboration with our vendor. While services remain stable, we are continuing to monitor the system closely to ensure ongoing reliability.

Read the full incident report →

Started May 28, 2025, 01:54 PM UTC· resolved May 28, 2025, 07:22 PM UTC

1. investigating May 28, 2025, 01:54 PM UTC

   The Class 4 SIP trunk core in Florida is unavailable at the moment. We are working to restore service there.

2. resolved May 28, 2025, 07:22 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 13, 2025, 04:23 PM UTC· resolved May 13, 2025, 07:40 PM UTC

1. investigating May 13, 2025, 04:23 PM UTC

   We are currently investigating this issue.

2. investigating May 13, 2025, 04:24 PM UTC

   We noticed a possible issue in the Florida datacenter and we’re working to remediate the issue. This has cleared without service impact.

3. monitoring May 13, 2025, 07:32 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved May 13, 2025, 07:40 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 01, 2025, 12:20 PM UTC· resolved May 01, 2025, 12:30 PM UTC

1. resolved May 01, 2025, 12:48 PM UTC

   Earlier this morning, prior to business hours, we experienced a brief event impacting the Zeus cluster that caused the customer portal to be temporarily unavailable for approximately 10 minutes. The issue self-recovered without manual intervention. Our engineering team was immediately engaged and is continuing to investigate the root cause. All portal services have since been fully restored and are operating normally.

Read the full incident report →

Started Apr 08, 2025, 12:36 AM UTC· resolved Apr 08, 2025, 03:34 AM UTC

1. investigating Apr 08, 2025, 12:36 AM UTC

   We are currently executing an emergency change to address the network issues affecting our Las Vegas infrastructure. During this change, there may be a brief loss of connectivity to the Las Vegas region, lasting only a few seconds. We expect minimal impact and appreciate your patience as we work to resolve the issue swiftly.

2. investigating Apr 08, 2025, 02:14 AM UTC

   We will be 503'ing the Las Vegas Apollo core while we perform configuration updates. Updates to follow.

3. resolved Apr 08, 2025, 03:34 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 04, 2025, 01:45 PM UTC· resolved Jun 11, 2025, 02:26 PM UTC

1. investigating Apr 04, 2025, 01:45 PM UTC

   Due to network connectivity issues between the New Jersey and Las Vegas data centers, we have decided to 503 the Las Vegas core. All traffic directed to the Las Vegas core will be redirected to the Florida core. If you have any questions or concerns, please reach out to [email protected] or call 1-833-VIIRTUE

2. identified Apr 04, 2025, 10:21 PM UTC

   We have identified the cause of the network issue and are working on a solution. Updates to follow.

3. identified Apr 05, 2025, 09:26 PM UTC

   We are continuing to work on a fix for this issue.

4. identified Apr 05, 2025, 09:28 PM UTC

   The network issue has been identified and corrected. Services in Las Vegas have been fully restored and are now operating normally.

5. identified Apr 07, 2025, 04:40 PM UTC

   The New Jersey data center experienced a network event earlier this morning that has since cleared. This event could have caused delays with the Xima integration. Updates to follow.

6. identified Apr 08, 2025, 01:29 AM UTC

   We will be temporarily redirecting traffic from the New Jersey data center to perform network tests. Updates to follow.

7. identified Apr 08, 2025, 02:15 AM UTC

   Service has been restored in the New Jersey data center.

8. identified Apr 08, 2025, 03:38 AM UTC

   Due to network issues at the Las Vegas data center, the Las Vegas core will remain offline. The server will be brought back online once the network has been tested and validated.

9. identified Apr 08, 2025, 01:11 PM UTC

   Service has been restored to the Las Vegas core.

10. identified Apr 08, 2025, 07:52 PM UTC

    Engineers are relocating devices away from the Las Vegas core to safely troubleshoot the network issues affecting this node. Updates will follow.

11. identified Apr 08, 2025, 09:02 PM UTC

    As part of our continued commitment to enhancing the stability, reliability, and overall performance of our platform, we are implementing an important change that will affect how data center locations are assigned for your domains. To reduce latency, minimize inter-region and geo-redundant calls, and streamline backend operations, we will be removing the option to manually select the data center location for your domains through the portal. Going forward, all domains will be automatically assigned to the optimal data center based on a combination of system performance metrics, geographic proximity, and redundancy best practices. This change is being made with your service experience in mind. By centralizing and automating data center assignment, we can significantly: • Improve overall service reliability and uptime • Reduce the risk of cross-region routing issues • Enhance failover efficiency and response times • Simplify management and reduce potential for configuration conflicts We understand that some users may have previously utilized manual data center selection to align with specific internal preferences. However, our new intelligent assignment model ensures a more consistent and high-performing experience for all customers without requiring manual input. This update will take effect today. There will be no impact to your existing domain functionality, and you will continue to receive the same level of service and support you expect from us. If you have any questions or concerns about this update, please don’t hesitate to reach out to our support team at [support contact info]. Thank you for your understanding and continued trust in our platform. We’re confident this improvement will contribute to an even more stable and seamless experience moving forward.

12. monitoring Apr 09, 2025, 05:09 PM UTC

    Incident Report – Interim RCA Date: April 8, 2025 Prepared by: Alberto Fernandez, Director Platform Engineering Overview: On Monday, April 7, 2025, we identified an issue impacting our New Jersey (NJ) data center, which was experiencing unusually high bandwidth utilization. Upon discovery, our engineering team took immediate action to expand the circuits in the NJ facility and initiated an in-depth investigation to determine the source of the increased traffic. Initial Response: We promptly performed sanity checks on the existing circuits to rule out any physical degradation or faults and proceeded to upgrade the circuits. Despite these mitigations, the bandwidth consumption remained abnormally high even with seemingly lower usage levels. Root Cause Analysis – Ongoing: Further investigation revealed that a network interface located in our Las Vegas (Vegas) data center was introducing packet drops. These packet losses negatively affected the SBus messaging system, which facilitates communication between all three of our cores. As a result, the SBus queue began to grow excessively, causing retransmissions and creating systemic side effects, including degraded performance in data centers that were otherwise healthy. Corrective Action: To prevent the Vegas site from continuing to destabilize the broader infrastructure, we made the strategic decision to take the Vegas core offline. Prior to this, we carefully migrated customers out of the Vegas environment to avoid service disruption. During the migration, however, we observed that some customers were manually reverting their configurations to route traffic back through the Vegas data center. These actions inadvertently nullified our remediation efforts. To preserve the integrity of the infrastructure and accelerate recovery, we made the decision to disable the portal feature allowing customers to manually select their data center location. This control will remain disabled while we restore all environments to a healthy and fully optimized state. Next Steps: We are actively developing a remediation and validation plan to address the networking issues within the Vegas data center. This plan includes a comprehensive round of testing to ensure long-term stability before reintegrating the Vegas site back into our production ecosystem. We strongly recommend that all customers whitelist the IP addresses of all three data centers within their firewall configurations. This ensures seamless service continuity in the event that we need to rehome services to a different location as part of our platform’s stability and automated recovery processes. Additionally, we advise against manually configuring phones, as this can lead to inconsistencies and potential service disruptions. Instead, we encourage the use of our automated provisioning systems and DNS SRV records, which are designed with built-in self-healing capabilities to enhance reliability and minimize downtime. Transparency and Forward Commitment: This document serves as an interim incident report and is not intended to serve as the final Root Cause Analysis (RCA). A formal RCA will be published once all issues are fully resolved and the affected infrastructure is confirmed to be stable. Our commitment to service excellence includes maintaining transparency during incidents. We will continue to provide timely updates and documentation to ensure our partners and customers are fully informed throughout the process. We appreciate your patience and understanding as we work diligently to restore full operational integrity across our environment.

13. resolved Jun 11, 2025, 02:26 PM UTC

    This incident has been resolved.

Read the full incident report →

Started Mar 11, 2025, 07:00 PM UTC· resolved Mar 11, 2025, 07:00 PM UTC

1. resolved Mar 11, 2025, 08:12 PM UTC

   Date: March 11, 2025 Time: 14:44:57 EST At approximately 14:44:57 EST on March 11, the NMS service on core2-nj (primary core processing service) experienced a crash, resulting in the temporary disruption of some active calls and device registrations. The service automatically restarted within seconds, and all impacted services recovered immediately. An open ticket is already in place with NetSapiens for further investigation into the root cause. We will provide updates as more information becomes available. Please reach out Support if you have any questions or concerns.

Read the full incident report →

Started Mar 01, 2025, 02:30 PM UTC· resolved Mar 01, 2025, 07:15 PM UTC

1. resolved Mar 01, 2025, 07:23 PM UTC

   ViiBE invoice generation did not initiate at 9:30 AM EST as expected. The issue was resolved at 2:15 PM EST and all invoices that were slated to be generated were processed successfully.

2. postmortem Mar 01, 2025, 07:23 PM UTC

   ## **Summary** On March 1st 2025, the scheduled invoice generation process for ViiBE failed to initiate at 9:30 AM EST as expected. The issue was identified and resolved by 2:15 PM EST, after which all invoices were successfully generated. ## **Impact** The failure to trigger invoice generation at the scheduled time resulted in a delay of approximately five hours and forty-five minutes. No invoices were lost, and all were eventually processed successfully. However, customers and internal stakeholders may have experienced delays in receiving invoice notifications. ## **Root Cause** Recent optimizations were made to streamline developers' local build processes by producing a single container image containing build artifacts for all services. This allowed faster local deployments to Kubernetes clusters. However, in the production environment, services are deployed using separate images, ensuring that only modified services are restarted. The difference in image deployment strategies led to a misconfiguration in the cron job definitions. While the build artifacts remained consistent across environments, the production deployment expected separate images per service. This mismatch resulted in the invoice generation cron job failing to trigger as expected. ## **Resolution** Once the issue was identified: 1. The cron job misconfiguration was corrected to align with the production deployment structure. 2. The affected images were updated, and the necessary services were redeployed. 3. A manual execution of the invoice generation process was performed to ensure all scheduled invoices were processed. ## **Preventive Measures** To mitigate the risk of similar issues in the future, the following actions will be taken: * **Enhance Configuration Validation**: Implement automated checks to validate cron job configurations before deployment. * **Consistency Between Local and Production Environments**: Standardize deployment approaches to minimize discrepancies between local development and production. * **Monitoring and Alerts**: Improve monitoring for scheduled jobs with proactive alerts when a cron job does not execute as expected. * **Pre-Deployment Testing**: Expand integration tests to cover scheduled jobs in a pre-production environment. ## **Conclusion** The issue was a result of a configuration oversight due to differences in build and deployment strategies between local development and production. The problem was identified and resolved with minimal long-term impact. Future preventive measures will help ensure more robust deployment practices and minimize the risk of similar failures.

Read the full incident report →

Started Jan 28, 2025, 03:07 PM UTC· resolved Jan 31, 2025, 02:14 PM UTC

1. investigating Jan 28, 2025, 03:07 PM UTC

   Our team is currently investigating reports of outdated voicemails appearing in the portal and completed calls displaying as active. We are working diligently to resolve the issue and will provide an update as soon as possible.

2. investigating Jan 29, 2025, 03:16 PM UTC

   We are continuing to work with our partners to resolve this issue—updates to follow.

3. identified Jan 30, 2025, 06:58 PM UTC

   A potential fix will be implemented tonight during scheduled maintenance.

4. resolved Jan 31, 2025, 02:14 PM UTC

   This incident has been resolved by restarting a few key services during a maintenance window.

Read the full incident report →

Started Jan 15, 2025, 04:37 PM UTC· resolved Jan 16, 2025, 02:07 PM UTC

1. investigating Jan 15, 2025, 04:37 PM UTC

   We are currently investigating a datacenter ISP carrier outage at the FL datacenter, which is affecting the ZEUS platform, including both the voice platform and management portal. Our team is working closely with the carrier to understand the root cause and resolve the issue as quickly as possible. We will continue to monitor the situation and provide updates as we learn more. Thank you for your patience and understanding during this time.

2. identified Jan 15, 2025, 04:43 PM UTC

   Viirtue Engineers have identified an issue with the Florida Zeus circuit. The web portal and voice traffic have been moved to the New Jersey datacenter. Users may experience issues with mobile app calls. Updates to follow.

3. monitoring Jan 15, 2025, 04:52 PM UTC

   Incident Update: The carrier is currently investigating network issues impacting the Florida region. In response, Viirtue teams have successfully migrated critical service traffic and will continue to monitor the situation throughout the remainder of the day.

4. identified Jan 15, 2025, 04:53 PM UTC

   Lumen has confirmed they are seeing a few outages in the Florida area. While the circuit looks be be stabilizing, traffic will remain pointed to the New Jersey Zeus circuit. Updates to follow.

5. identified Jan 15, 2025, 07:49 PM UTC

   Lumen is working to resolve the outages affecting the Zeus Florida circuit. Updates to follow.

6. identified Jan 15, 2025, 09:09 PM UTC

   Zeus users may experience issues downloading call recordings from the portal, especially for calls older than 4 days. Updates to follow.

7. resolved Jan 16, 2025, 02:07 PM UTC

   Incident has been resolved. Details from carrier: Root Cause: An equipment failure in Tampa, FL impacted services. Fix Action: Services were migrated to an alternate path and the affected card was replaced in Forest Park, GA, thus restoring services to a stable state. This service impact is over. Lumen Network Operations Center (NOC) management will be reviewing this outage for any additional analysis or discovery. In some cases, you may need to reset your services locally at your equipment. If a service issue continues after that reset, please contact the Lumen Repair Center for help.

8. postmortem Jan 16, 2025, 03:44 PM UTC

   The Zeus Florida circuit experienced significant packet loss due to a fiber issue in Lumen's network, which affected customers in the Florida region. Upon identifying the root cause of the issue, the Viirtue team promptly redirected all traffic to the Zeus New Jersey datacenter to mitigate the impact. Lumen resolved the fiber issue early Thursday morning, after which the Viirtue team thoroughly tested the circuit. Once confirmed stable, traffic was successfully redirected back to the Zeus Florida datacenter.

Read the full incident report →

Started Jan 13, 2025, 07:52 PM UTC· resolved Jan 13, 2025, 09:37 PM UTC

1. investigating Jan 13, 2025, 07:52 PM UTC

   Viirtue Engineers are currently investigating an issue affecting the Core2-nj Apollo server. Updates to follow.

2. monitoring Jan 13, 2025, 08:05 PM UTC

   A solution has been implemented, and our engineering team will continue to monitor the system closely. Further updates will be provided as needed.

3. resolved Jan 13, 2025, 09:37 PM UTC

   This incident has been resolved. More details will be listed in the post-mortem section of this incident.

Read the full incident report →

Started Dec 02, 2024, 09:27 PM UTC· resolved Dec 03, 2024, 04:49 AM UTC

1. identified Dec 02, 2024, 09:27 PM UTC

   The Netsapiens PBX Usage Report which is accessible through ViiBE is missing both seat counts and mobile connect counts. We will trigger a reprocessing this evening to ensure those values are populated. We will follow up with more details once the data has been populated. Existing data for November may be inaccessible until the process completes. Updates to follow.

2. resolved Dec 03, 2024, 04:49 AM UTC

   The Netsapiens PBX Usage Report for the month of November 2024 should now how all values populated correctly. We thank you for your patience.

3. postmortem Dec 03, 2024, 04:49 AM UTC

   **Incident Overview:** Following recovery efforts from Hurricane Milton, a database restoration inadvertently overwrote a critical configuration required for the NetSapiens PBX Usage Report. **Root Cause Analysis:** The restoration process performed by our technology partner did not include steps to preserve or verify the specific configuration relied upon by the NetSapiens PBX Usage Report. This oversight resulted in the configuration being overwritten during the recovery. **Actions Taken:** 1. **Collaboration with Technology Partner:** * The incident was reviewed with our technology partner to ensure mutual understanding of the issue and its impact. 2. **Update to Standard Operating Procedures \(SOPs\):** * The technology partner has updated their SOPs to include a specific step for preserving and restoring the NetSapiens PBX Usage Report configuration during any future database restoration. 3. **Confirmation of SOP Updates:** * We have received confirmation from the technology partner that these SOP updates have been implemented effectively. **Future Mitigation:** In the unlikely event that a similar database restoration is required, the updated SOPs ensure that the configuration critical to the NetSapiens PBX Usage Report will not be missed, significantly reducing the risk of recurrence. This incident underscores the importance of robust documentation and verification in restoration procedures, and we are confident that the corrective actions taken will prevent similar issues in the future.

Read the full incident report →

Started Dec 02, 2024, 04:56 PM UTC· resolved Dec 02, 2024, 04:56 PM UTC

1. resolved Dec 02, 2024, 04:56 PM UTC

   The QoS score discrepancies observed in the portal at approximately 10:30 AM Eastern have been resolved. Please note that these scores were inaccurate due to an issue with the analytics server and did not reflect actual call quality, which remained unaffected during this time.

Read the full incident report →