Is Servd down?

We are in the process of cycline them out. Some project components may be unavailable for a few seconds while we do so.

We've now resolved the incident. Thanks for your patience.

Between 09:36 and 09:40 UTC, our US-EAST-6 cluster received a large-scale DDoS attack which briefly caused the cluster's load balancers to return 502 Bad Gateway responses to a subset of projects hosted in US-EAST-6. At 09:37 UTC, our automated Cloudflare-backed DDoS protections activated and began blocking malicious traffic. Meanwhile, the clusters load balancers scaled up and error rates dropped back to normal levels. In response, we've permanently increased the baseline capacity of the clusters load balancer and are prioritising measures to better absorb the initial shock of similar DDoS attacks in the future. Apologies for the interruption.

We have noted increased error rates on a subset of projects caused by connectivity issues with projects' redis instances.

The issue has now been fully resolved. The issue was caused by a deployment misconfiguration which prevented PHP from being able to communicate with redis. We are currently in the process of refactoring the use of Redis (which has changed is licensing to prevent commercial use) within projects deployed to the Servd platform. The first step of this change is to create a set of feature gates within our deployment mechanism which allow us to flip a project between Redis and the proposed replacement (currently DragonflyDB). Yesterday we released these feature gates to all projects' deployment mechanisms with the default still set to use Redis. We were therefore expecting no change to any projects at this time, but would have the ability to flip the feature gates on a per project basis in the future. This was tested on multiple internal projects before rolling out generally. At 01:30 BST we were alerted to elevated 500 errors from multiple projects. Upon inspection these projects seemed to be having issues communicating with their Redis instances. We assumed this must be related to the feature gate changes that had rolled out earlier in the day. These projects had all had their subscriptions renewed over the past 30 minutes which was the apparent trigger. During subscription renewals we perform a 'partial sync' on projects - syncing updates related to changes to project subscriptions without deploying any settings changes made in the Servd dashboard. This relies on taking a snapshot of the project's previous sync's deployment values and merging those with any changes due to the subscription. This resulted in the new feature gates having no default value set - the sync was executed with them as empty values. Our deployment scripts also contained logic that checked for the value of the feature gates to direct traffic to either redis or its replacement. This logic failed due to the empty value of the feature gate (it expected a specific boolean value) which led to the bad configuration being deployed. To fix the problem we ran through all impacted projects and executed a 'full sync' on each of them. This ensured a non-empty value was applied for the feature gates and allowed a correct configuration to be deployed. We also subsequently updated the deployment scripts to handle the empty value elegantly to ensure the problem does not repeat for further project renewals. The total impact on projects varied in time as we ran through them from start to finish. We have also adjusted our testing methodology for any deployment script updates which include new values, to include testing with partial syncs to ensure they work as expected.

At 00:57 UTC, Servd's US-EAST-1 cluster received a large DDoS attack which lasted 30 seconds. At which point our Cloudflare-backed DDoS protections activated and ended the attack. In the interim, we saw elevated 502 responses as our infrastructure quickly scaled up to handle the load. Apologies for any interruptions to your projects. For those looking to insulate themselves from such attacks we recommend employing [Servd's Private Load Balancer addon](https://servd.host/docs/private-load-balancer-addon).