Nebula Outage History

Started Jul 09, 2025, 10:24 AM UTC· resolved Jul 10, 2025, 08:03 AM UTC

1. investigating Jul 09, 2025, 10:24 AM UTC

   Our NOC are investigating reports of degraded performance across various elements of our platform, including calls, softphone apps and load speeds with our online dashboards. We will provide a further update in 30 minutes.

2. identified Jul 09, 2025, 10:40 AM UTC

   The affected traffic is being rerouted and we're beginning to see performance in those areas improve. We are continuing to monitor and will provide a further update in the next 30 minutes.

3. monitoring Jul 09, 2025, 10:51 AM UTC

   Service has been restored in most areas and continues to improve in some niche cases. Our Team are still monitoring the incident and will mark this as resolved after a successful period. More information will be provided later via Post Mortem on this status.

4. identified Jul 09, 2025, 11:45 AM UTC

   We're investigating further reports of similar issues and will provide a further update within 30 minutes

5. monitoring Jul 09, 2025, 12:07 PM UTC

   Normal service has resumed again and our team continue to closely monitor the situation.

6. identified Jul 09, 2025, 12:56 PM UTC

   We are seeing degraded performance in some areas and working on the fix as we speak.

7. monitoring Jul 09, 2025, 01:17 PM UTC

   A fix has been implemented and we are monitoring the situation.

8. resolved Jul 10, 2025, 08:03 AM UTC

   This is confirmed as resolved, and a Post Mortem will be provided ASAP via this status page.

9. postmortem Jul 11, 2025, 09:22 AM UTC

   ### **What Happened** At approximately 11:20 am BST on 9th July, we identified a sudden and unprecedented surge in traffic through our network, which resulted in a build up of messages entering the platform. This created a rapid slowdown of elements of the platform that rely on this technology; namely call authentication, softphone app status and chat functions, and dashboard navigation. ### **Impact** While we have rigorous monitoring and load balancing protections in place, the rate at which this occurred meant that our usual redundancy measures were not able to sufficiently redirect all offending traffic and this ultimately led to a degradation of service which was seen across a significant number of accounts. Our team worked throughout the day to both mitigate the impact on our end users, and also investigate and identify the root cause. We systematically suspended various elements of our network while we investigated the issue, and each time were periodically restarting affected servers which cleared the backlog instantly, however shortly after each restart the traffic returned to unprecedented levels via different channels. Unfortunately this led to a number of ‘false positives’ on the root cause, leading our team to incorrectly and prematurely confirm a resolution via our status page. We ultimately identified the root cause as a specific customer’s misconfiguration of an ancillary service to our main product line. Unlike with our own software, the protections around a complex integration with this third-party service rely in part on safeguards implemented by the other party; in this case, we found there was insufficient rate limiting on their side, which allowed an inflated level of traffic into the Nebula platform. This traffic came from various origins and made it harder for our team to quickly identify the root cause. **Resolution** After identifying the origin of the issue, we immediately suspended all services surrounding the account and integration. Once incoming traffic returned to normal levels, the backlog was immediately cleared and normal service resumed within minutes. We then worked with the service provider to establish the cause on their end, and implemented further checks within our own platform to prevent it recurring. **Next Steps** We have since begun a comprehensive review of all aspects of the connecting infrastructure, and will be implementing further improvements to allow us to better monitor and identify any similar issues in future. This also includes a more comprehensive review of our complete technology stack to ensure we are sufficiently protected for other cases of a similar nature. We would like to sincerely apologise for the disruption caused, and thank you for your patience and understanding.

Read the full incident report →

Started Jul 08, 2025, 08:54 AM UTC· resolved Jul 08, 2025, 11:48 AM UTC

1. identified Jul 08, 2025, 08:54 AM UTC

   Our team have identified an issue with the Users page of customer dashboards not loading. We are deploying a fix and should be resolved within the next 30 minutes.

2. monitoring Jul 08, 2025, 09:01 AM UTC

   We've completed the update and the users page should be operational again. We'll continue to monitor and will mark as resolved later today.

3. resolved Jul 08, 2025, 11:48 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 20, 2025, 07:48 AM UTC· resolved Jun 20, 2025, 03:22 PM UTC

1. investigating Jun 20, 2025, 07:48 AM UTC

   Our team are investigating reports of Hot Desking devices failing to log in, and new configuration updates failing to upload to Yealink handsets. We will provide a further update within 30 minutes.

2. identified Jun 20, 2025, 08:24 AM UTC

   Our Team have identified the issue and are working on a solution. WIll provide a further update within 30 minutes.

3. monitoring Jun 20, 2025, 08:30 AM UTC

   Our team have implemented a fix and can see handsets are now provisioning successfully. We will continue to monitor the situation for the remainder of the day.

4. resolved Jun 20, 2025, 03:22 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 09, 2025, 09:55 AM UTC· resolved Jun 12, 2025, 09:18 AM UTC

1. identified Jun 09, 2025, 09:55 AM UTC

   Our engineering team are investigating reports of slow load and login times on our online dashboards, and desktop softphone applications. Calls and physical handsets are not currently affected. We will provide another update within 1 hour. Customers may also experience delays in webhooks and voicemail notifications coming through.

2. identified Jun 09, 2025, 10:48 AM UTC

   Our team continue to investigate the issue. We are hearing reports users are being logged out of the softphone applications and then experiencing issues with logging back in. We are looking at this with urgent priority and will provide a further update in the next hour.

3. identified Jun 09, 2025, 11:33 AM UTC

   We are continuing to work on a fix for this issue.

4. identified Jun 09, 2025, 12:40 PM UTC

   Our engineers are still investigating the issue, and will provide a further update within 1 hour.

5. identified Jun 09, 2025, 02:03 PM UTC

   We are still investigating this issue and will provide a further update within 1 hour.

6. identified Jun 09, 2025, 03:02 PM UTC

   Our team are in the process of diverting the affected traffic to additional resources, and will be monitoring for further improvements.

7. monitoring Jun 09, 2025, 05:04 PM UTC

   We've implemented a fix and are seeing services restoring to normal. We will continue to monitor this incident over the next 24 hours and provide further updates if necessary.

8. resolved Jun 12, 2025, 09:18 AM UTC

   This incident has been resolved.

9. postmortem Jun 16, 2025, 02:14 PM UTC

   ### **What Happened** At approximately 9:30 am BST on 9th June, we began receiving reports of dashboards being slow to load, alongside degraded performance on softphone applications and wallboards. We also observed that certain backend processes were delayed. **Desk phones and** **live calls were not affected at any point during the incident.** Upon investigation, our engineering team identified that our primary database was operating at maximum capacity. This was not caused by a sudden traffic spike, but rather a gradual increase in the volume of frequently accessed "hot data" over time. Once this threshold was reached, the resource-intensive swapping process created a system-wide slowdown. ### **Impact** The direct impact on customers was a significant slowdown across the platform. To protect the core service, we temporarily paused several non-essential backend processes like advanced wallboards. While we have extensive monitoring, once the threshold was crossed, the volume of delayed processes created an immediate and significant impact before automated systems could fully mitigate it. **Resolution** Our Engineering and DevOps teams implemented two hugely significant updates to resolve the issue and permanently improve the platform’s overall performance. First, we deployed a software fix that overhauled our API infrastructure, optimising and reducing the queries sent to the database. This was completed by 5:00 pm on 9th June. After a brief period of clearing the backlog, normal service resumed. Second, we performed a zero-downtime infrastructure upgrade overnight on 10th June, significantly increasing the server memory and processing power. **Next Steps** We are confident these changes provide a permanent and significant improvement to platform resilience. To ensure this issue does not reoccur, we are taking the following steps: 1. **Enhanced Monitoring:** We have implemented more specific monitoring, based on the characteristics of this incident, to help us identify and prevent similar issues far earlier. 2. **Proactive Optimisation:** Our engineering team is conducting a comprehensive review of our codebase to identify and improve other potentially inefficient processes. 3. **Capacity Planning:** We are refining our forecasting to ensure our infrastructure always scales well ahead of demand. We thank you for your patience and understanding as we worked to resolve this and apologise for the disruption caused.

Read the full incident report →

Started May 20, 2025, 01:56 PM UTC· resolved May 21, 2025, 02:15 PM UTC

1. identified May 20, 2025, 01:56 PM UTC

   We're investigating reports PDD and calls not connecting. Will provide a further update in 10 minutes.

2. monitoring May 20, 2025, 02:03 PM UTC

   We've identified an issue with Google Cloud Platform, however they have confirmed the majority of issues are resolved and we can see call traffic returning to normal. We will continue to monitor and provide further updates as soon as they are available. Link here: https://status.cloud.google.com/incidents/SXRPpPwx2RZ5VHjTwFLx

3. monitoring May 20, 2025, 02:11 PM UTC

   Call traffic has stabilised and no further reports are coming through, however we will continue to monitor this incident and close within 24 hours if no further updates.

4. monitoring May 20, 2025, 02:11 PM UTC

   We are continuing to monitor for any further issues.

5. monitoring May 20, 2025, 03:53 PM UTC

   To provide a further update on this incident so far: Google Cloud Platform acknowledged an incident on it's status page at around 12pm BST on Monday, however after investigation our NOC were satisfied that the affected portions of it's infrastructure had no adverse affect on our platform or services. At around 14:48pm BST, we became aware of reports of issues seen some customers, which aligned with the issues described by GCP, and coincided with the temporary loss of services with their online portals. Our DevOps team were in the process of escalating this incident with GCP, however shortly after at around 15:00pm BST we begun to see call traffic return to normal levels. We will continue to closely monitor both the GCP status and our own internal traffic reports, and will close this incident as fully resolved after a further 24 hours.

6. resolved May 21, 2025, 02:15 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 05, 2025, 03:00 PM UTC· resolved May 05, 2025, 03:55 PM UTC

1. investigating May 05, 2025, 03:00 PM UTC

   We have found the root-cause of the issue. Fixing in-progress Next update in 5minutes..

2. investigating May 05, 2025, 03:08 PM UTC

   We are continuing to investigate this issue.

3. investigating May 05, 2025, 03:20 PM UTC

   We are still experiencing issues with outbound calls. The issue is under investigation. Next update shortly.

4. investigating May 05, 2025, 03:45 PM UTC

   We are continuing to investigate this issue.

5. resolved May 05, 2025, 03:55 PM UTC

   The issue is now resolved. All the services are up and running now.

6. postmortem May 07, 2025, 02:11 PM UTC

   **What Happened** At 03:10 PM BST, our monitoring systems detected connectivity issues between the database and Nebula platform which occurred due to an unexpected network refresh. As the network reconnected, the database encountered an authentication error. This caused a connection issue for inbound and outbound calls for 90mins. **Impact** The inbound and outbound calls were impacted and couldn't be connected through the platform. **Resolution** We took immediate corrective measures, which involved identifying the root-cause of the issue \(DNS related\). Services were restored within 90mins with no further delays encountered in the call flow. A subsequent 24-hour monitoring period ensured no recurrence, allowing us to investigate and confirm the nature of the fault. In the wake of the incident, we have implemented additional DNS and connectivity health-checks which run every minute. **Next Steps** The implemented corrective and preventative measures will significantly reduce the risk of similar incidents in the future. We are committed to providing reliable and high-quality voice to our partners and customers and will continue to review and improve our platform's resilience and performance.

Read the full incident report →

Started Mar 25, 2025, 12:36 PM UTC· resolved Mar 26, 2025, 12:58 PM UTC

1. investigating Mar 25, 2025, 12:36 PM UTC

   Our engineers are currently investigating reports of issues answering inbound calls with some handset and softphone app users on isolated accounts. We'll provide another update within 20 minutes.

2. identified Mar 25, 2025, 12:50 PM UTC

   We have identified the issue and are working to implement a fix ASAP. With provide a further update in 20 mins.

3. monitoring Mar 25, 2025, 01:06 PM UTC

   After a temporary period of degraded performance, all affected traffic has now been rerouted. Any devices that are still not receiving calls should reconnect again after re-registration (reboot handset, or refresh/ close and reopen soft phone application). We will continue monitoring and provide an update within the next hour if any further issues reported.

4. monitoring Mar 25, 2025, 02:01 PM UTC

   We are seeing previously affected devices connecting successfully after re-registering (reboot handset or refresh apps to manually re-register), and will close this incident after 24 hours of monitoring.

5. resolved Mar 26, 2025, 12:58 PM UTC

   After successful monitoring we have marked this incident as resolved. A RFO will be posted within the next 48 hours.

6. postmortem Mar 28, 2025, 11:34 AM UTC

   **What Happened** On March 25, 2025, at approximately 12:30 PM, a server handling phone registration data experienced an isolated incident. This caused a temporary loss of registration for handsets and apps connected to that server, preventing them from receiving incoming calls. The issue lasted less than one hour, with all affected devices regaining full service by 1:30 PM. **Impact** Only devices actively registered to the affected server at the time of the crash were impacted. New registration attempts were unaffected, as our backup servers handled those requests seamlessly. Most devices re-registered automatically within 15 minutes, though some took up to an hour depending on their registration renewal cycle. For accounts with multiple handsets, some devices may have stayed operational if connected to unaffected servers. **Resolution** The impacted server was restored, and service resumed without requiring manual intervention from users. Our redundancy systems ensured new registrations continued uninterrupted throughout the incident. **Next Steps** We’re committed to keeping our platform reliable. To prevent this from happening again, we’ve added enhanced monitoring to catch and mitigate such issues even earlier. We apologise for any disruption and appreciate your understanding.

Read the full incident report →

Started Mar 11, 2025, 09:32 AM UTC· resolved Mar 12, 2025, 11:30 AM UTC

1. investigating Mar 11, 2025, 09:32 AM UTC

   Our engineers are investigating reports of delay and silence on calls, affecting a subset of customers. Further updates will be provided within the next 20 minutes.

2. investigating Mar 11, 2025, 09:42 AM UTC

   We are continuing to investigate the issue, but note this is also extended to the ability to answer some calls.

3. identified Mar 11, 2025, 09:47 AM UTC

   Our engineers have identified the issue and are working towards a fix ASAP. Next update within the next 20 mins.

4. identified Mar 11, 2025, 10:12 AM UTC

   We are continuing to work on a fix and will provide a further update ASAP.

5. identified Mar 11, 2025, 10:37 AM UTC

   Work is ongoing and a further update will be provided within the next 30 mins.

6. monitoring Mar 11, 2025, 10:50 AM UTC

   Traffic has been returning to normal levels in most areas of our network. We'll continue to monitor and provide further updates within one hour.

7. monitoring Mar 11, 2025, 11:52 AM UTC

   Our NOC will continue to monitor the incident over the next 24 hours, before marking as resolved.

8. monitoring Mar 12, 2025, 11:04 AM UTC

   After a period of successful monitoring we are marking this incident as resolved. A Post Mortem will be published within 24 hours.

9. resolved Mar 12, 2025, 11:30 AM UTC

   This incident has been resolved.

10. postmortem Mar 13, 2025, 09:36 AM UTC

    At 09:10am UTC, our monitoring systems detected a sudden and substantial increase in SIP traffic directed towards our platform. Analysis revealed that the surge originated from an unprecedented volume of SIP requests, generated by malfunctioning equipment in a specific area of our infrastructure designed for large-scale deployments. The excessive packet rate created a cascading effect; the initial overload caused delays in call processing, in turn triggering further retries from the affected endpoints, exacerbating the problem. This flood of traffic pushed our servers to full capacity within minutes, and the Nebula platform immediately re-routed traffic to unaffected servers allowing around 70% of our services to operate unaffected. While the platform has resilient fault-tolerance processes, the remaining servers \(and consequentially any end users connecting to those servers at random\) were unfortunately disrupted while we worked to address the underlying cause. This resulted in performance degradation including increased call latency, dropped calls, and potential service unavailability for users connecting to the affected infrastructure. We took immediate corrective measures, which involved identifying, isolating and then blocking traffic from the affected zones. Service was restored for the users connected to the affected servers as fast as we reasonably could without running the risk of further disruption. A subsequent 24 hour monitoring period ensured no recurrence, allowing us to investigate and confirm the nature of the fault. In the wake of the incident, we have significantly improved our monitoring systems to isolate and address similar symptoms before they cause any disruption, and as an additional precaution have now implemented several methods to identify the characteristic and frequency of those types of packets and isolate that traffic from the voice element of our infrastructure. The implemented corrective and preventative measures will significantly reduce the risk of similar incidents in the future. We are committed to providing reliable and high-quality voice to our partners and customers, and will continue to review and improve our platform's resilience and performance.

Read the full incident report →

Started Jan 24, 2025, 12:52 PM UTC· resolved Jan 27, 2025, 08:42 AM UTC

1. investigating Jan 24, 2025, 12:52 PM UTC

   Our engineers are currently investigating reports of issues with apps and dashboards loading, and call connection. We will provide next update within 20 minutes.

2. investigating Jan 24, 2025, 12:53 PM UTC

   We are continuing to investigate this issue.

3. identified Jan 24, 2025, 01:01 PM UTC

   The issue has been identified and we're beginning to see traffic return to normal levels. We will continue to monitor and confirm once satisfied the incident is resolved.

4. monitoring Jan 24, 2025, 02:11 PM UTC

   All services had resumed normal operations at 13:06pm today. Our teams are continuing to monitor all areas of the platform and will mark this incident as resolved following successful review over the weekend.

5. resolved Jan 27, 2025, 08:42 AM UTC

   After a successful period of monitoring over the weekend our engineers are satisfied that this incident is resolved.

6. postmortem Jan 27, 2025, 10:49 AM UTC

   On January 24th at 12:40pm we rolled out a security patch which was needed to address a newly identified vulnerability with one of our provisioning services. This affected only our supporting systems such as the website and dashboard, and took around 8 minutes to complete, with a few minutes thereafter to clear the backlog. During this time there was no significant impact on our telephony system, although some customers did report difficulty initiating new calls due to the authentication process.

Read the full incident report →

Started Dec 10, 2024, 02:00 PM UTC· resolved Dec 11, 2024, 08:45 AM UTC

1. identified Dec 10, 2024, 02:00 PM UTC

   This morning our engineers begun to investigate reports of calls not reaching users running our CallSwitch One and PulseHD apps mobile apps on iOS. We have identified the issue and are working to resolve with urgency, and will provide an update within the next 4 hours.

2. monitoring Dec 10, 2024, 03:42 PM UTC

   Our engineers have implemented a fix and we're seeing calls receiving successfully on iOS devices, for both CallSwitchOne and PulseHD apps. We're continuing to monitor the situation and will close this incident as resolved in due course.

3. resolved Dec 11, 2024, 08:45 AM UTC

   After a successful monitoring period this incident is now resolved.

Read the full incident report →

Started Dec 03, 2024, 05:59 PM UTC· resolved Dec 06, 2024, 10:04 AM UTC

1. investigating Dec 03, 2024, 05:59 PM UTC

   We are currently receiving a small number of calls concerning phones in a hunt group continuing to ring after the call is answered. We are currently investigating the issue and will issue an update as soon as possible.

2. identified Dec 04, 2024, 12:27 PM UTC

   Our engineers have found the root cause of the issue and are currently testing a patch to fix it. This will be rolled out in off-peak hours.

3. monitoring Dec 05, 2024, 12:49 PM UTC

   A fix was implemented out of hours yesterday and initial testing seems to indicate this issue has been resolved. We will continue monitoring and update further in due course.

4. resolved Dec 06, 2024, 10:04 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Nov 15, 2024, 09:39 AM UTC· resolved Nov 15, 2024, 10:04 AM UTC

1. investigating Nov 15, 2024, 09:39 AM UTC

   We are currently investigating this issue.

2. identified Nov 15, 2024, 09:42 AM UTC

   We have identified a minor issue which is causing some users to not be able to access the dashboard. It is currently being reset and will be available very shortly.

3. resolved Nov 15, 2024, 10:04 AM UTC

   This incident has been resolved.

Read the full incident report →