HetrixTools Outage History

Started Nov 01, 2026, 06:27 AM UTC· resolved Nov 01, 2026, 06:34 AM UTC

1. investigating Nov 01, 2026, 06:27 AM UTC

   We're currently investigating some misfired "Agent Data Warning" notifications for some of our server monitoring agents.

Read the full incident report →

Started Oct 21, 2026, 01:07 PM UTC· resolved Oct 21, 2026, 07:19 PM UTC

1. identified Oct 21, 2026, 01:07 PM UTC

   We've now taken the rest of our front-end web offline, awaiting the release of the PHP patches. https://www.php.net/archive/2026.php#2026-10-20-1 Uptime Monitoring and Blacklist Monitoring remain fully working, so you will be notified of events.

2. identified Oct 21, 2026, 03:05 PM UTC

   We have now applied the PHP patches, and we're restoring the functionality of our platform. We'll update you here with more information soon.

3. monitoring Oct 21, 2026, 03:11 PM UTC

   Services are now back online and fully functional. We're monitoring the situation further to make sure no unexpected issues appear.

4. resolved Oct 21, 2026, 07:19 PM UTC

   This incident is now resolved.

Read the full incident report →

Started Oct 21, 2026, 10:10 AM UTC· resolved Oct 21, 2026, 04:17 PM UTC

1. identified Oct 21, 2026, 10:10 AM UTC

   We've currently disabled our White Label Pages as a precaution, pending the release of the critical PHP vulnerability patches: https://www.php.net/archive/2026.php#2026-10-20-1

2. monitoring Oct 21, 2026, 10:15 AM UTC

   We've put a preliminary defensive mechanism in place, and the White Label pages are now back online. We'll continue to monitor the situation as it progresses.

3. monitoring Oct 21, 2026, 11:26 AM UTC

   We're still monitoring the situation. The only outage for our White Label Pages was 5 minutes, from 10:10 UTC to 10:15 UTC. Further updates will be provided as needed in this incident.

4. identified Oct 21, 2026, 12:49 PM UTC

   We are taking down the White Label Pages again while we wait for the PHP patches to be released. Further info will be available here about this incident.

5. resolved Oct 21, 2026, 04:17 PM UTC

   This incident is now resolved. The total outage of our White Label Pages was roughly 2 hours. Related incident: https://status.hetrixtools.com/incidents/nnlfj1y45tzy

Read the full incident report →

Started Oct 14, 2026, 08:21 AM UTC· resolved Oct 22, 2026, 11:42 AM UTC

1. identified Oct 14, 2026, 08:21 AM UTC

   We've identified an issue with Microsoft's SNDS API where the Blocklist data is returned empty. Our system is keeping its old data in such cases. This means the Microsoft SNDS Blocklist isn't receiving any new data right now. We're waiting for Microsoft to fix this issue on their end. The SNDS Red/Yellow lists are not affected; only the SNDS Blocklist is.

2. monitoring Oct 17, 2026, 06:27 AM UTC

   This incident is still ongoing; there's still no sign of improvement on the Microsoft SNDS API side. We'll continue to monitor this situation as it progresses.

3. resolved Oct 22, 2026, 11:42 AM UTC

   This incident has now been resolved.

Read the full incident report →

Started Jun 18, 2026, 02:40 PM UTC· resolved Jun 19, 2026, 03:03 PM UTC

1. investigating Jun 18, 2026, 02:40 PM UTC

   We're investigating a significant increase in API failures when fetching blacklisting data from the Microsoft SNDS platform. This may cause delayed blacklisting/delisting notices on the Microsoft SNDS blacklists.

2. identified Jun 19, 2026, 07:57 AM UTC

   The issue has been identified: Microsoft SNDS silently changed how its API responds to data requests, breaking most of our SNDS data sync integration. We're currently working on implementing a fix for this issue and forcing a new data sync after.

3. monitoring Jun 19, 2026, 09:29 AM UTC

   A fix has now been implemented, and we're monitoring the situation further. The SNDS system is fully working as expected now.

4. resolved Jun 19, 2026, 03:03 PM UTC

   The issue has now been resolved. The SNDS system is fully working as expected now.

Read the full incident report →

Started May 05, 2026, 09:26 PM UTC· resolved May 05, 2026, 11:12 PM UTC

1. identified May 05, 2026, 09:26 PM UTC

   We've been internally alerted of a higher-than-usual fail rate for our Uptime Monitoring system. The DENIC domain registry is currently having issues, causing most of the .DE domains to not resolve. Our system is currently detecting these uptime monitors as being down, as it should be, since they are not resolving properly. Sources: https://status.denic.de/pages/incident/592577eab611ce1e0d00046f/69fa60ef9d12f5057a974f38 https://news.ycombinator.com/item?id=48027897 https://www.reddit.com/r/de_EDV/s/vh0wekNZ3p This issue is out of our hands, and our system is working as expected. We're posting this here so that our users are informed of what is happening with their .DE uptime monitors. We'll continue to monitor the situation as it progresses.

2. resolved May 05, 2026, 11:12 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 30, 2026, 01:12 PM UTC· resolved Apr 30, 2026, 02:00 PM UTC

1. resolved Apr 30, 2026, 01:12 PM UTC

   At around 13:12 UTC, we noticed an increase in errors with our Server Monitoring Metrics API. After further investigation, our techs discovered that one of our back-end nodes started experiencing issues and was not ingesting all monitoring metrics as expected. The issue has been fully resolved by 14:00 UTC. The issue has affected about 2.7% of our Server Monitoring Agents, which may have had issues posting their metrics to our platform.

Read the full incident report →

Started Mar 30, 2026, 07:39 PM UTC· resolved Mar 30, 2026, 11:43 PM UTC

1. investigating Mar 30, 2026, 07:39 PM UTC

   We're currently investigating issues that are affecting our Blacklist Monitoring Queue.

2. identified Mar 30, 2026, 09:04 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Mar 30, 2026, 09:45 PM UTC

   A fix has now been implemented, and we're slowly restarting the microservices powering the Blacklist Monitoring Queue.

4. monitoring Mar 30, 2026, 09:47 PM UTC

   The queue has a big backlog to process, so it will take some time for it to catch up and process monitors in real time, as it does when healthy. We'll provide further updates here as this process advances.

5. monitoring Mar 30, 2026, 10:48 PM UTC

   The queue backlog is now at 50% of its level when processing was resumed. We'll follow up with another update once it's fully caught up.

6. resolved Mar 30, 2026, 11:43 PM UTC

   The backlog has now been fully processed, and the Blacklist Monitoring Queue is now running healthy as expected.

Read the full incident report →

Started Mar 18, 2026, 12:20 PM UTC· resolved Mar 18, 2026, 03:14 PM UTC

1. identified Mar 18, 2026, 12:20 PM UTC

   We've been observing an increased rate of 520 HTTP errors from HEAD requests to websites behind Cloudflare. Cloudflare has recently acknowledged the issue on their end and is working on a fix: https://www.cloudflarestatus.com/incidents/kxvggpg7kwx5 In the meantime, you can fix the issue right away in your HetrixTools account by changing the "HTP Method" from "HEAD" to "GET" for any of your website monitors (by editing the monitor's settings): https://docs.hetrixtools.com/wp-content/uploads/2022/10/image.png You can check if you're affected by this Cloudflare issue by checking your Location Fail Log (https://docs.hetrixtools.com/location-fail-log/) and looking for 520 HTTP errors.

2. resolved Mar 18, 2026, 03:14 PM UTC

   Cloudflare has updated the status of their incident to resolved. We're noticing that the number of 520 HTTP errors from HEAD requests has gone down. This incident is now resolved.

Read the full incident report →

Started Dec 24, 2025, 05:00 AM UTC· resolved Dec 24, 2025, 09:00 AM UTC

1. resolved Dec 24, 2025, 05:00 AM UTC

   The server hosting our Documentation has suffered a critical disk failure. Recovery has taken a bit more than initially expected. The functionality of our Documentation has been impaired for approximately 4 hours and 9 minutes. It has now been fully restored and recovered.

Read the full incident report →

Started Dec 06, 2025, 08:17 PM UTC· resolved Dec 06, 2025, 08:17 PM UTC

1. resolved Dec 06, 2025, 08:17 PM UTC

   This RBL has become unresponsive for a few days now. We have now removed it from our system. If your IPs or domains were blacklisted here, they will be marked as delisted after the next automatic checkup. We'll reactivate the RBL if/when it comes back online.

Read the full incident report →

Started Dec 06, 2025, 03:11 PM UTC· resolved Dec 16, 2025, 04:41 PM UTC

1. identified Dec 06, 2025, 03:11 PM UTC

   We're observing a significant increase in API failures when fetching blacklisting data from the Microsoft SNDS platform. They have recently (4 Dec 2025) performed some changes in how their API works, and we believe these errors might be related to that. Our system has been configured for such scenarios and will fallback onto older/stale data if it cannot fetch fresh blacklisting data from the SNDS API. This may cause delayed blacklisting/delisting notices on the Microsoft SNDS blacklists. Unfortunately, this issue is out of our hands, and we hope they'll fix their API soon so our system can resume regular monitoring of the Microsoft SNDS blacklists. In the meantime, we're keeping an eye on the situation as it progresses.

2. identified Dec 09, 2025, 08:38 AM UTC

   We're still seeing very high 400/500 HTTP error and timeout rates from the Microsoft SNDS API. Our system is currently able to update the SNDS blacklisting data once a day, down from the usual once per hour under normal circumstances. Our tech department is making ongoing efforts to bring this frequency down further. We'll keep you posted here with further updates as we have them.

3. identified Dec 10, 2025, 10:08 PM UTC

   Our techs have now reduced the update frequency down to every 3 hours. We're still seeing an unusually high number of errors from the Microsoft SNDS API.

4. resolved Dec 16, 2025, 04:41 PM UTC

   Based on tests we've run over the past few days, we've concluded that updating the SNDS data more often than once every 12 hours will result in too many errors from the SNDS API. We're currently sticking to this update frequency until these errors are addressed on their end.

Read the full incident report →

Started Nov 18, 2025, 11:34 AM UTC· resolved Nov 18, 2025, 04:34 PM UTC

1. identified Nov 18, 2025, 11:34 AM UTC

   We're aware that some of our components behind Cloudflare are experiencing issues at this time: https://www.cloudflarestatus.com/incidents/8gmgl950y3h7 This currently affects our services as follows: Our Blacklist Monitoring service remains unaffected. Our Uptime Monitoring service continues to process a high number of detected outages, so notifications may be delayed as it processes through its backlog. The Server Agent Metrics system is impaired, as it is behind Cloudflare.

2. identified Nov 18, 2025, 02:13 PM UTC

   Unfortunately, the outage is still ongoing at this time. We'll continue to monitor the situation as it progresses. Further info about this outage can be found on Cloudflare's status page: https://www.cloudflarestatus.com/incidents/8gmgl950y3h7

3. monitoring Nov 18, 2025, 02:43 PM UTC

   We're seeing signs that the Cloudflare service is almost fully functional again, with most of the websites behind Cloudflare being seen as UP again by our Uptime Monitoring system. We'll continue to monitor the situation.

4. monitoring Nov 18, 2025, 02:49 PM UTC

   The Server Agent Metrics endpoint behind Cloudflare is still experiencing a partial loss (~15%) of incoming data.

5. monitoring Nov 18, 2025, 04:11 PM UTC

   We are still observing a relatively small number of monitoring agents (~5%) unable to access our Cloudflare API endpoint to post their metric data. We'll continue to monitor the situation as it progresses.

6. resolved Nov 18, 2025, 04:34 PM UTC

   Cloudflare has now resolved the incident. We're still monitoring and hands-on in case issues resurface.

Read the full incident report →

Started Nov 02, 2025, 06:00 AM UTC· resolved Nov 02, 2025, 12:00 PM UTC

1. identified Nov 02, 2025, 06:00 AM UTC

   We’ve identified an issue in our Linux Server Monitoring Agent that occurs when Daylight Saving Time ends (in autumn). If the server where the agent is installed is not configured to use the UTC timezone, the agent will fail to collect metrics during the repeated (backward) hour. This only affects the following versions of our Linux Server Monitoring Agent: 2.3.1 2.3.2 2.3.3 2.3.4 The issue stems from how the agent scheduler uses the server's time to run the agent every one minute; a scheduler that won't fire up when the hour turns backwards. A solution has been implemented for this with version 2.3.5 of our Linux Server Monitoring Agent. You can easily update your agent with the following command: wget -4 -qO- https://raw.githubusercontent.com/hetrixtools/agent/master/hetrixtools_update.sh | sudo bash

2. resolved Nov 02, 2025, 12:00 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 20, 2025, 05:32 PM UTC· resolved Oct 20, 2025, 06:36 PM UTC

1. investigating Oct 20, 2025, 05:32 PM UTC

   We are currently investigating an issue where some Email notifications might be sent multiple times due to ongoing AWS Issues.

2. identified Oct 20, 2025, 05:50 PM UTC

   We have identified the issue and are currently working on a fix for it. Impact: Daily/Weekly/Monthly Reports and Blacklist Monitoring Notifications may be delayed while we fix the issue.

3. identified Oct 20, 2025, 06:07 PM UTC

   We implemented a fix, and the Email Queue is now functioning properly and catching up on its backlog. We're monitoring it to make sure everything works as expected.

4. resolved Oct 20, 2025, 06:36 PM UTC

   The Email Queue has caught up with its backlog and is now functioning as expected.

Read the full incident report →

Started Oct 20, 2025, 07:03 AM UTC· resolved Oct 20, 2025, 10:36 AM UTC

1. investigating Oct 20, 2025, 07:03 AM UTC

   We are currently investigating connectivity issues with AWS infrastructure in North America. This can cause some parts of our platform to load up slowly or time out.

2. identified Oct 20, 2025, 07:36 AM UTC

   The incident has been identified and confirmed by AWS, as per their status page: https://health.aws.amazon.com/health/status We're working on mitigating the impact on our end while they work on solving the root cause of the problem.

3. monitoring Oct 20, 2025, 09:23 AM UTC

   We're seeing no more errors related to the AWS outage. We're still monitoring the situation to make sure that our systems are fully recovered.

4. resolved Oct 20, 2025, 10:36 AM UTC

   This incident has now been fully resolved.

Read the full incident report →

Started Jul 14, 2025, 10:17 PM UTC· resolved Jul 15, 2025, 10:50 PM UTC

1. identified Jul 14, 2025, 10:17 PM UTC

   Cloudflare's DNS servers (1.1.1.1 and 1.0.0.1) seem to be having major issues: https://www.cloudflarestatus.com/incidents/28r0vbbxsh8f https://x.com/Newspicel/status/1944880481704796373 This issue affects our Server Monitoring Agent installed on servers that use these DNS servers. Since these servers cannot resolve any hostnames at this time, our Server Monitoring Agent cannot send any metrics to our platform. Resolution: Use diverse DNS servers on your servers (i.e., a mixture of Cloudflare and Google DNS servers), or wait for Cloudflare to fix its DNS servers so that your server regains full DNS resolution capabilities.

2. monitoring Jul 14, 2025, 10:29 PM UTC

   We're seeing traffic coming back towards normal limits, as Cloudflare implemented a fix for its DNS servers. We'll continue to monitor the situation.

3. resolved Jul 15, 2025, 10:50 PM UTC

   We're now seeing traffic back to normal, even though Cloudflare hasn't yet marked the issue as resolved on their status page.

Read the full incident report →