Sitejet incident

DDoS led to sporadic errors

Resolved after 14 minutes, on March 5, 2025.

postmortem Mar 13, 2025, 02:43 PM UTC

Due to the high number of concurrent requests with a long timeout, a middleware reached its connection limit and had to reject database requests, thus causing 502 errors within the User Interface. In the meantime, we heavily increased the connection capacity on that middleware to avoid such an issue in the future. Despite this unsuccessful attack, all websites remained fully operational, experiencing no downtime and maintaining their usual level of efficiency.

Started
Mar 05, 2025, 10:35 AM UTC
Resolved
Mar 05, 2025, 10:49 AM UTC
Duration
14m
Detected by Pingoru
Mar 05, 2025, 10:35 AM UTC

Update timeline

  1. resolved Mar 05, 2025, 10:30 AM UTC

    Today, our services were challenged by a substantial Distributed Denial of Service (DDoS) attack, causing sporadic 502 Bad Gateway errors in our User Interfaces between 11:35 and 11:49 (CET). Our automated countermeasures swiftly identified and blocked thousands of malicious IPs taking part in the attack, significantly minimizing its impact. While these efforts greatly mitigated the situation, some 502 errors were unavoidable. We apologize for the inconvenience and disruption this caused our users and are committed to enhancing our security posture to better handle such challenges in the future.

  2. postmortem Mar 13, 2025, 02:43 PM UTC

    Due to the high number of concurrent requests with a long timeout, a middleware reached its connection limit and had to reject database requests, thus causing 502 errors within the User Interface. In the meantime, we heavily increased the connection capacity on that middleware to avoid such an issue in the future. Despite this unsuccessful attack, all websites remained fully operational, experiencing no downtime and maintaining their usual level of efficiency.

Your product runs on more than Sitejet

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →