Happydance Outage History

Started Oct 16, 2026, 10:49 AM UTC· resolved Oct 16, 2026, 11:08 AM UTC

1. investigating Oct 16, 2026, 10:49 AM UTC

   We’re aware of an issue affecting some careers websites and our team is investigating as a priority. We’ll keep you updated here regularly as we learn more.

2. monitoring Oct 16, 2026, 11:06 AM UTC

   A fix has been applied and we’re monitoring to make sure everything is stable. We’ll keep you updated here regularly until this is fully resolved.

3. resolved Oct 16, 2026, 11:08 AM UTC

   This issue is now resolved. We’re sorry for any disruption caused. A post-incident review will be shared with affected customers.

Read the full incident report →

Started Sep 01, 2026, 02:52 PM UTC· resolved Sep 01, 2026, 04:25 PM UTC

1. investigating Sep 01, 2026, 02:52 PM UTC

   We're aware of an incident that may have affected our US-based sites. Sites may have experienced intermittent downtime and our team are actively investigating the root cause and working to ensure full stability.

2. investigating Sep 01, 2026, 04:02 PM UTC

   We're continuing to investigate and are working closely with the team to identify the root cause. We'll share our next update shortly.

3. resolved Sep 01, 2026, 04:25 PM UTC

   This incident has now been resolved. Thanks for your patience while we worked through it.

4. postmortem Sep 03, 2026, 08:48 AM UTC

   ## What happened On September 1, several Happydance hosted career websites had trouble loading search and job listing pages. Some visitors saw pages load slowly or fail to return results during this window. The issue is fully resolved, and all affected career websites are operating normally. ## What caused it A group of career websites run on the same underlying search infrastructure to keep job search fast and consistent. During the affected window, one career website received an unusually high volume of automated traffic to its search pages. That surge used up more capacity on that infrastructure than expected, and once capacity was exhausted, search requests from other career websites running on the same infrastructure also began timing out. Those other websites' own servers and databases were unaffected and continued running normally throughout. ## What we did * Increased the capacity of the search infrastructure to absorb the extra load. * Monitored search and listing pages until performance returned to normal across all affected career websites. ## What we're doing next * Improving how we detect and filter automated traffic before it reaches search infrastructure. * Reviewing monitoring on this infrastructure so we catch a spike like this sooner. We're sorry for the disruption this caused. If you have questions about how this affected your career website specifically, reach out to your Happydance support contact.

Read the full incident report →

Started Feb 11, 2026, 12:43 AM UTC· resolved Feb 11, 2026, 08:49 AM UTC

1. investigating Feb 11, 2026, 12:43 AM UTC

   We’re currently investigating an unplanned availability issue affecting one of our US1 servers. At 12:37 AM GMT on Wednesday, 11 February 2026, Azure reported that the underlying virtual machine became temporarily unavailable. This was detected by Azure’s platform monitoring. At this stage: The issue is platform-side No action is required from customers Azure is actively assessing the root cause and recovery status We’re closely monitoring the situation and will provide updates here as soon as more information becomes available.

2. monitoring Feb 11, 2026, 01:36 AM UTC

   We’re seeing monitoring services gradually restore to normal levels following the earlier unplanned availability event affecting the US1 server. Recovery is still in progress, and some monitoring checks may continue to appear delayed or inconsistent while the platform fully stabilises. We’ll continue to monitor closely and will provide further updates if anything changes.

3. resolved Feb 11, 2026, 08:49 AM UTC

   The earlier unplanned availability issue affecting the US1 server has now been fully resolved. Services have been stable for some time, and monitoring confirms that all systems are operating normally. This incident was caused by a temporary platform-level event reported by Azure. No further impact is expected. We will continue routine monitoring as part of our standard operational procedures. If you experience any lingering issues, please contact support.

4. postmortem Feb 11, 2026, 02:27 PM UTC

   ## Incident Overview On 11 February 2026 between 00:29 and 01:43 GMT, customer websites hosted on US1 \(Central US\) experienced service disruption following an unexpected virtual machine availability event. Azure Resource Health reported the virtual machine as unavailable during this period. While the platform began recovering, the operating system did not restore cleanly, resulting in prolonged service instability before normal operation resumed. No data loss occurred during the incident. ## Root Cause Azure reported an unplanned platform-level availability event affecting the underlying infrastructure hosting the US1 virtual machine. During the event: * The US1 virtual machine became temporarily unavailable * The VM underwent a platform-level recovery/restart process * The Windows operating system did not initialise cleanly * The server entered an unstable state during startup, delaying full service restoration Although the Azure platform recovered, the operating system exhibited signs of instability post-restart. This behaviour is consistent with corruption or inconsistency introduced during abrupt platform-level interruption. Operating system and attached data disks were preserved throughout the event. No database or customer data corruption was identified. ## Incident Response * The availability issue was detected immediately via monitoring and Azure Resource Health alerts * Engineering confirmed the impact was isolated to workloads hosted on US1 * Platform health and VM state were reviewed through Azure diagnostics * Services gradually began restoring as the VM stabilised * Once service stability was verified, the incident was marked as resolved Following recovery, a deeper assessment identified residual operating system instability risk. ## Mitigation Steps Immediate actions taken: * Verified integrity of hosted websites and application services * Confirmed no data loss or disk corruption * Reviewed Azure Service Health notifications for Central US Preventative action \(in progress\): * Planned migration of impacted websites to a new compute resource * Provisioning of a clean virtual machine instance * Controlled migration of workloads to eliminate residual OS-level risk * Decommissioning of the affected compute resource once migration is complete This approach ensures long-term stability rather than relying on an OS instance that experienced an abnormal recovery state. ## Conclusion This incident was caused by an unplanned Azure platform availability event that resulted in virtual machine interruption. While Azure restored platform availability, the operating system entered an unstable recovery state, extending service disruption. All services have been stable since 01:43 GMT on 11 February 2026. As a precautionary measure, we are migrating affected workloads to a new compute resource to eliminate any potential residual instability and further strengthen service resilience. We will continue to monitor platform health and take proactive steps to minimise the impact of future infrastructure-level events

Read the full incident report →

Started Jan 25, 2026, 10:38 PM UTC· resolved Jan 26, 2026, 08:40 AM UTC

1. investigating Jan 25, 2026, 10:38 PM UTC

   We’re currently aware of an issue impacting some websites hosted on US1. Our engineering team is investigating the cause and working to restore full service as quickly as possible. We’ll provide further updates here as more information becomes available. Thank you for your patience.

2. monitoring Jan 25, 2026, 11:43 PM UTC

   The issue affecting websites hosted on US1 has now stabilised and services are starting to return to normal. We’re continuing to monitor the platform closely to ensure full recovery and stability. A further update will be provided if required. Thank you for your patience.

3. resolved Jan 26, 2026, 08:40 AM UTC

   The issue affecting websites hosted on US1 has been resolved and services are operating normally. We’ll continue to monitor the platform to ensure ongoing stability. Thank you for your patience.

4. postmortem Jan 26, 2026, 09:20 AM UTC

   ## Incident overview On Sunday evening, customer websites hosted on US1 \(Central US\) experienced service disruption following an unexpected virtual machine restart. The restart coincided with an urgent Azure preventive maintenance event affecting underlying network infrastructure in the Central US region. During the reboot process, the affected server became stuck in a Windows restart/update loop, delaying service recovery. ## Root cause Azure identified a degradation in a network device connected to one or more virtual machines in the Central US region and initiated urgent preventive repair work to avoid wider unplanned failures. As part of this event, virtual machines may be automatically rebooted or migrated by the Azure platform. In this case: * The US1 virtual machine underwent an unexpected reboot * During startup, Windows entered a restart/update loop * The loop prevented normal service startup and temporarily blocked standard recovery actions No data loss occurred. Operating system and data disks were retained throughout the event. ## Incident response * The issue was detected shortly after the unexpected restart * Affected services were isolated to US1-hosted workloads * Engineering investigated both platform health and recovery options * The Windows restart loop resolved naturally once the underlying platform state stabilised * Services began returning to normal and the platform was placed under monitoring Once stability was confirmed, the incident was marked as resolved. ## Mitigation steps * Continued platform monitoring following recovery * Verification of service availability and application health * Review of Azure Service Health notifications related to Central US maintenance * Confirmation that no data corruption or configuration loss occurred ## Conclusion This incident was caused by Azure-initiated urgent preventive maintenance combined with a Windows restart loop during VM recovery. While the initial restart was unplanned from an application perspective, it aligns with Azure’s advance notification of urgent infrastructure repair in the region. All services are now operating normally. We will continue to monitor platform health and take proactive steps to reduce the impact of future infrastructure-level maintenance events.

Read the full incident report →

Started Dec 05, 2025, 09:08 AM UTC· resolved Dec 05, 2025, 10:31 AM UTC

1. investigating Dec 05, 2025, 09:08 AM UTC

   We are aware of an ongoing Cloudflare incident affecting the Cloudflare services. This may temporarily impact career sites. There is no known impact to live customer traffic at this time. All sites continue to operate normally. Our team is monitoring the situation closely and assessing any potential effects. We will provide further updates as Cloudflare publishes more information or if we detect any customer-facing impact.

2. monitoring Dec 05, 2025, 09:20 AM UTC

   Cloudflare has implemented a fix and monitoring the results. Career sites continue to operate normally.

3. resolved Dec 05, 2025, 10:31 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Nov 18, 2025, 11:49 AM UTC· resolved Nov 18, 2025, 07:07 PM UTC

1. investigating Nov 18, 2025, 11:49 AM UTC

   Current Status: Intermittent Connectivity Issues We are currently experiencing intermittent connectivity problems affecting some customers. The issue has been traced to an upstream Cloudflare service disruption. Our team is monitoring the situation closely and working to minimise any impact. We will provide further updates as Cloudflare restores normal service. Thank you for your patience.

2. investigating Nov 18, 2025, 12:22 PM UTC

   Cloudflare is aware of, and investigating an issue which impacts multiple customers: Widespread 500 errors. We are working to understand the full impact and mitigate this problem. More updates to follow shortly.

3. monitoring Nov 18, 2025, 12:28 PM UTC

   We are seeing services recover, but customers may continue to observe higher-than-normal error rates as we continue remediation efforts.

4. monitoring Nov 18, 2025, 01:50 PM UTC

   Cloudflare are continuing to work on restoring service for application services customers.

5. monitoring Nov 18, 2025, 02:28 PM UTC

   Cloudflare are continuing to work on a fix for this issue.

6. monitoring Nov 18, 2025, 02:48 PM UTC

   Cloudflare update: A fix has been implemented and we believe the incident is now resolved. We are continuing to monitor for errors to ensure all services are back to normal.

7. monitoring Nov 18, 2025, 03:54 PM UTC

   Cloudflare are continuing to focus on restoring service post-fix. They are mitigating several issues that remain post-deployment.

8. monitoring Nov 18, 2025, 04:15 PM UTC

   Cloudflare update: Bot scores will be impacted intermittently while we undergo global recovery. We will update once we believe bot scores are fully recovered.

9. monitoring Nov 18, 2025, 04:41 PM UTC

   Cloudflare continue to see errors and latency improve but still have reports of intermittent errors. The team continues to monitor the situation as it improves, and looking for ways to accelerate full recovery.

10. monitoring Nov 18, 2025, 05:20 PM UTC

    Cloudflare continue to monitor the system through recovery and are seeing errors and latency return to normal levels. A full post-incident investigation and details about the incident will be made available asap. Patsy

11. monitoring Nov 18, 2025, 07:07 PM UTC

    Cloudflare services are currently operating normally. We are no longer observing elevated errors or latency across the network. Our engineering teams continue to closely monitor the platform and perform a deeper investigation into the earlier disruption, but no configuration changes are being made at this time. At this point, it is considered safe to re-enable any Cloudflare services that were temporarily disabled during the incident. We will provide a final update once our investigation is complete.

12. resolved Nov 18, 2025, 07:07 PM UTC

    This incident has been resolved.

13. postmortem Nov 19, 2025, 09:44 AM UTC

    ## **Incident overview** On 18 November 2025, Cloudflare experienced a global service disruption that caused HTTP 500 errors before traffic reached our infrastructure. During this period, visitors to your website may have seen Cloudflare’s error page instead of reaching the hosted origin. Your hosting platform remained stable. The issue occurred entirely within Cloudflare’s network, and normal service was restored once Cloudflare completed its fixes. The outage began at 11:20 UTC, core traffic stabilised at 14:30 UTC, and all Cloudflare systems were fully restored by 17:06 UTC. ## **Root cause** The incident was triggered by an internal Cloudflare configuration error. A database permissions change caused Cloudflare’s Bot Management system to generate a malformed configuration file. This file doubled in size and exceeded a strict memory limit inside Cloudflare’s global proxy layer. When deployed across their network, the proxy failed and returned HTTP 500 responses to end users. There was no attack, DDoS, or malicious activity involved. ## **Incident response** Cloudflare’s engineering teams moved quickly once error rates spiked: * They investigated the surge in 500 errors and initially suspected a large-scale DDoS due to unusual traffic behaviour. * They traced the issue to the malformed Bot Management feature file. * They stopped propagation of the incorrect file and replaced it with a stable version. * They restarted affected proxy services globally to restore normal traffic flow. Our platform remained fully operational throughout the incident. The issue was entirely upstream at Cloudflare, which prevented traffic from reaching our environment. Because access is secured through Cloudflare, the origin couldn’t be reached until Cloudflare restored normal service. ## **Mitigation steps** ### **Immediate actions** Cloudflare applied several emergency measures to stabilise traffic: * Halted generation and distribution of the malformed configuration file. * Deployed a known-good version across all global edge locations. * Restarted core proxy systems to clear the failure state. * Implemented temporary bypasses for dependent services to reduce secondary impact. ### **Long-term prevention** Cloudflare has begun work on permanent improvements, including: * Strengthening validation for all internally generated configuration files. * Adding additional global kill switches for safety-critical components. * Improving memory-handling and error-recovery logic within their proxy modules. * Enhancing database query controls to prevent malformed output. * Reviewing failure modes across Bot Management and other proxy-related systems. ## **Conclusion** This outage was caused entirely by a configuration failure within Cloudflare’s Bot Management system. Your website hosting, infrastructure, and the Happydance platform remained stable and were not the source of the interruption. Cloudflare resolved the issue the same afternoon and has committed to platform-wide improvements to prevent recurrence. We’ll continue to monitor Cloudflare’s remediation progress and ensure your environment stays protected. If you’d like help sharing this internally or need further clarification, we’re here to support you. ## **Reference** Full technical summary from Cloudflare: [**https://blog.cloudflare.com/18-november-2025-outage/**](https://blog.cloudflare.com/18-november-2025-outage/)

Read the full incident report →

Started Oct 29, 2025, 04:00 PM UTC· resolved Oct 30, 2025, 12:44 AM UTC

1. investigating Oct 29, 2025, 04:00 PM UTC

   We are currently investigating reports of various sites having connectivity issues and will update further once investigation has concluded

2. investigating Oct 29, 2025, 04:12 PM UTC

   We are continuing with our investigation, Azure is still reporting no issues or loss of service. But we have narrowed down the issue to any customers using our legacy Azure Firewall (Azure Front Door)

3. identified Oct 29, 2025, 04:45 PM UTC

   Microsoft Azure has reported an ongoing DNS-related issue impacting availability of certain Azure services. While most of our environments remain unaffected, we have confirmed that customers using our legacy Azure Firewall (Azure Front Door) may experience site availability issues. We are continuing to monitor the situation closely and are working to confirm stability following Microsoft’s mitigation actions. Further updates will be provided as new information becomes available.

4. identified Oct 29, 2025, 05:02 PM UTC

   Microsoft Azure has confirmed the issue is related to a wider DNS disruption and is actively performing failover and recovery actions across their Azure Front Door infrastructure. Our monitoring continues to show that only customers using our legacy Azure Firewall (Azure Front Door) may experience intermittent availability or connectivity issues. All other environments remain fully operational. We will continue to monitor the situation closely and provide a further update once additional information becomes available from Microsoft.

5. identified Oct 29, 2025, 05:33 PM UTC

   Microsoft Azure have given a further update on the issue (last updated at 17:18 UTC on 29 October 2025): Starting at approximately 16:00 UTC, we began experiencing Azure Front Door (AFD) issues resulting in a loss of availability of some services. We suspect that an inadvertent configuration change as the trigger event for this issue. We are taking two concurrent actions where we are blocking all changes to the AFD services and at the same time rolling back to our last known good state. We have failed the portal away from AFD to mitigate the portal access issues. Customers should be able to access the Azure management portal directly. We do not have an ETA for when the rollback will be completed, but we will update this communication within 30 minutes or when we have an update. https://azure.status.microsoft/en-gb/status

6. identified Oct 29, 2025, 06:06 PM UTC

   Azure last updated at 17:51 UTC on 29 October 2025 confirming there is still no ETA on rollback and globally all services behind Azure Front Door & Their DNS are offline. Any customers that have completed the migration to Cloudflare, these issues are not effecting your services.

7. monitoring Oct 29, 2025, 08:16 PM UTC

   Latest update from Microsoft at 19:57 UTC on 29 October 2025 We initiated the deployment of our ‘last known good’ configuration, which has now successfully been completed. Customers may have begun to see initial signs of recovery. We are currently recovering nodes and routing traffic through healthy nodes, and as we make progress in this workstream, customers will continue to see improvement. At this stage, we anticipate full mitigation within the next four hours as we continue to recover nodes. This means we expect recovery to happen by 23:20 UTC on 29 October 2025. We will provide another update on our progress within two hours, or sooner if warranted.

8. monitoring Oct 30, 2025, 12:43 AM UTC

   Azure services have been restored. We have confirmed and reached out to the customers whose sites were affected. We will be sending out PIR reports in the next 72 hours with the details of the incident.

9. resolved Oct 30, 2025, 12:44 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 22, 2025, 11:34 AM UTC· resolved Oct 22, 2025, 04:14 PM UTC

1. investigating Oct 22, 2025, 11:34 AM UTC

   Customers hosted on the US1 shared instance are currently experiencing extended downtime following a scheduled maintenance window. A Windows update on the underlying infrastructure took significantly longer than expected to complete, resulting in temporary service unavailability for hosted websites. Our team has confirmed that the update has now reached the final stages and the instance is expected to return to service shortly. We will continue to monitor progress closely and provide an update as soon as the instance is fully restored. Impact: US1 Shared Instance: Unavailable All other shared and dedicated instances: Operational

2. monitoring Oct 22, 2025, 12:09 PM UTC

   The US1 shared instance remains in the final stages of maintenance following the earlier Windows update. While services are gradually restoring, customers may still experience intermittent availability or performance issues as system processes complete and stability is verified. Our team continue to monitor the environment closely and are working to ensure full service restoration as soon as possible. Another update will be provided once all systems are confirmed stable.

3. monitoring Oct 22, 2025, 12:30 PM UTC

   The US1 shared instance has begun its automatic restart following the completion of system updates. This reboot is part of the final maintenance phase and is expected to restore all services shortly. Our team is monitoring the restart and will provide a further update once the instance is fully online and validated.

4. monitoring Oct 22, 2025, 12:57 PM UTC

   We’re continuing to work on restoring full service for the US1 shared instance following this morning’s maintenance. Our team is taking additional recovery steps to ensure the environment is stable before bringing all sites fully back online. Services may remain intermittently unavailable during this process. We’ll provide another update once restoration has completed and systems are confirmed stable.

5. monitoring Oct 22, 2025, 01:43 PM UTC

   Services on the US1 shared instance have been fully restored and are operating normally. Our team will continue to monitor system performance closely to ensure continued stability and optimal operation. A final update will be provided once monitoring confirms sustained normal performance.

6. monitoring Oct 22, 2025, 01:43 PM UTC

   We are continuing to monitor for any further issues.

7. resolved Oct 22, 2025, 04:14 PM UTC

   Service on the US1 shared instance has been fully restored and operating normally. The issue was caused by a Windows update on one server that entered a rollback sequence during scheduled maintenance, resulting in temporary unavailability. All other environments completed the same update successfully. The affected instance was recovered from a validated backup, and post-restoration monitoring confirms stability across all services. We appreciate your patience during this maintenance window.

Read the full incident report →

Started Apr 01, 2025, 11:12 AM UTC· resolved Apr 03, 2025, 07:24 AM UTC

1. investigating Apr 01, 2025, 11:12 AM UTC

   We are currently investigating an issue with Azures US1 Central Network effecting customers. We will provide more information as we identify the cause.

2. investigating Apr 01, 2025, 11:16 AM UTC

   Failover for all US1 customers has commenced.

3. investigating Apr 01, 2025, 11:35 AM UTC

   We have successfully migrated all US1 customers to our failover region US West 1

4. monitoring Apr 01, 2025, 11:35 AM UTC

   We will continue to monitor the US1 Region and move customers back once stable

5. monitoring Apr 01, 2025, 12:24 PM UTC

   We are continuing to see issues with the US network as a whole and are trying to mitigate these for our customers

6. monitoring Apr 01, 2025, 04:21 PM UTC

   We are holding off putting customers back to US central until we get confirmation from Azure that there is no further impact on our services. Initial Investigation looks to be a network hardware failure in their central datacenter.

7. monitoring Apr 02, 2025, 01:22 PM UTC

   All websites have now been restored to Central US (US1). Services are operating normally, and we continue to monitor the server closely to ensure stability.

8. resolved Apr 03, 2025, 07:24 AM UTC

   This incident has been resolved.

Read the full incident report →