CloudCannon Outage History
CloudCannon is up right nowPingoru has recorded 50 incidents for CloudCannon: 16 in the last 12 months and 0 in the last 90 days.
Incidents, newest first
Minor March 21, 2025 Docker login error on builds 32m
Started Mar 21, 2025, 02:15 AM UTC· resolved Mar 21, 2025, 02:47 AM UTC
- investigating Mar 21, 2025, 02:15 AM UTC
We are currently investigating a docker login error causing build failures.
- identified Mar 21, 2025, 02:31 AM UTC
The issue has been identified and a fix is being implemented.
- resolved Mar 21, 2025, 02:47 AM UTC
The issue has been resolved and all sites that failed during the period have been rebuilt. Thank you for your patience
Minor March 3, 2025 Elevated 503 errors 1h 46m Hosting (sites.cloudcannon.com)
Started Mar 03, 2025, 10:32 PM UTC· resolved Mar 04, 2025, 12:19 AM UTC
- investigating Mar 03, 2025, 10:32 PM UTC
We are currently investigating elevated 503 errors for sites hosted on CloudCannon.
- monitoring Mar 03, 2025, 11:04 PM UTC
Affected servers have been refreshed and we are monitoring ongoing performance.
- resolved Mar 04, 2025, 12:19 AM UTC
This incident has been resolved.
Minor February 24, 2025 Increased rate of 503 errors 1h 8m Hosting (sites.cloudcannon.com)
Started Feb 24, 2025, 08:51 AM UTC· resolved Feb 24, 2025, 09:59 AM UTC
- investigating Feb 24, 2025, 08:51 AM UTC
We are currently investigating this issue.
- identified Feb 24, 2025, 09:01 AM UTC
The issue has been identified to be a redis cache under load. We are working to increase resources allocated.
- monitoring Feb 24, 2025, 09:21 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Feb 24, 2025, 09:59 AM UTC
This incident has been resolved.
- postmortem Feb 25, 2025, 09:40 PM UTC
At 9:34 pm \(24th February NZT\), our servers reported a Redis out-of-memory error. This error caused issues across different areas of the application, with the primary issue being 503 errors when uncached sites were requested. Our team started an investigation at 9:44 pm, and by 10 pm, we had identified the Redis issue. Our immediate solution was to increase the memory allocated to the Redis machine. A larger machine was made available at 10:14 pm. Once the servers returned to normal, the previously errored clients started reconnecting. This created a large influx of connections, increasing the load on our servers. We temporarily scaled up our servers to account for the extra load. Our team continued to monitor performance and deemed the service was back to normal by 10:57 pm. CloudCannon maintained the increased memory allocation until we could implement a longer-term solution. Upon reviewing our Redis database, we found a number of databases that were not expiring correctly. Some databases are now expiring correctly, and existing items have been updated with the same TTL. The remaining databases which contained larger key s have been migrated to use S3 files. This creates a sustainable solution that allows memory to be self-maintained. At 4:47 pm \(25th February NZT\), we updated the Redis server to its original size.
Minor February 11, 2025 Syncing downtime 37m App (app.cloudcannon.com)
Started Feb 11, 2025, 07:53 PM UTC· resolved Feb 11, 2025, 08:30 PM UTC
- investigating Feb 11, 2025, 07:53 PM UTC
We are currently investigating an issue where saves and Git syncs are not working.
- investigating Feb 11, 2025, 07:53 PM UTC
We are continuing to investigate this issue.
- identified Feb 11, 2025, 07:59 PM UTC
We have discovered the issue and a fix is being implemented.
- monitoring Feb 11, 2025, 08:04 PM UTC
A fix has been implemented and we are continuing to monitor the results. We are working through impacted sites to ensure they are all recovered correctly.
- monitoring Feb 11, 2025, 08:18 PM UTC
The sync queue is at 60 sites and going down. Thank you for your patience.
- resolved Feb 11, 2025, 08:30 PM UTC
The queue is cleared and operations are back to normal. We are investigating additional alarms and infrastructure changes to prevent the issue in the future.
Minor February 3, 2025 Builds failing to start 27m
Started Feb 03, 2025, 03:47 PM UTC· resolved Feb 03, 2025, 04:15 PM UTC
- monitoring Feb 03, 2025, 03:47 PM UTC
We have implemented a fix and builds are running again. There is a queue of builds running now. We will continue to monitor the issue until all are completed.
- resolved Feb 03, 2025, 04:15 PM UTC
Builds have returned to normal and the queue has been cleared. Thank you for your patience with this issue.
Minor January 30, 2025 Issues syncing 2h 36m App (app.cloudcannon.com)
Started Jan 30, 2025, 06:24 PM UTC· resolved Jan 30, 2025, 09:00 PM UTC
- identified Jan 30, 2025, 06:24 PM UTC
We are working on fixing an issue that is causing syncs to fail due to low disk space.
- monitoring Jan 30, 2025, 06:47 PM UTC
A fix has been implemented and syncs are now running again. We are reviewing sites that were impacted by the downtime and working through the backlog of syncs.
- monitoring Jan 30, 2025, 07:31 PM UTC
We have resolved the issues and syncing has returned to normal. There are a handful of sites that have become stuck due to this downtime, we are continuing to work on fixing these sites.
- resolved Jan 30, 2025, 09:00 PM UTC
We have resolved all pending issues on the sites impacted. Thank you for your patience. Contact support if you have any questions or concerns.
Minor December 26, 2024 Form Submission Processing Delays 1h 1m
Started Dec 26, 2024, 07:33 PM UTC· resolved Dec 26, 2024, 08:35 PM UTC
- investigating Dec 26, 2024, 07:33 PM UTC
Our form submission processing infrastructure is running behind. We are investigating the issue.
- identified Dec 26, 2024, 07:46 PM UTC
We have identified and fixed the issue. The backlog of form submissions is being processed now.
- monitoring Dec 26, 2024, 07:56 PM UTC
The backlog of data is being processed now. We are continuing to monitor it.
- resolved Dec 26, 2024, 08:35 PM UTC
The incident has been resolved and all backlog has been processed.
- postmortem Jan 22, 2025, 08:27 PM UTC
### Summary of the Incident On December 27, 2024, at 6:29 AM NZT, a customer, reported that form submissions on their website were not being processed, a situation confirmed to have started three days earlier. Escalation of the incident followed promptly: * **6:41 AM**: The issue was escalated to CTO. * **8:17 AM**: CTO escalated the ticket to the on-call engineer, who initiated the investigation. ### Root Cause Analysis The investigation revealed that the failure originated from an AWS ECS task update for the `hooks-worker` service: * The ECS `hooks-worker` task failed to initialize following an AWS update/restart. * The failure was due to a `CannotPullContainerError`, indicating that the required image in the Elastic Container Registry \(ECR\) was no longer available due to lifecycle policy limitations. Relevant log: ``` CannotPullContainerError: pull image manifest has been retried 1 time(s): failed to resolve ref ...: not found ``` ### Immediate Actions Taken * A new deployment of the `hooks-worker` pipeline using an updated image was approved and executed. * ECS tasks were scaled from 1 to 4 to process the backlog of approximately 7,770 messages. * By **8:56 AM**, the backlog was cleared, and the ECS tasks were scaled back to 1. * Additional services potentially impacted by similar issues, were redeployed with updated task definitions by **11:18 AM**. ### Long-Term Improvements 1. ECR Repository Lifecycle Policies * **Issue**: The ECR lifecycle policy retained insufficient versions \(30 images\), leading to premature cleanup due to the additional ARM image builds. * **Actions**: * Increased the ECR lifecycle policy retention by threefold. * Planned a review of the ARM build process to reduce noise and optimize repository usage. 1. Alarm Noise Reduction * **Issue**: Alerts for the oldest message age were received but overlooked due to excessive noise in the alert channel. * **Actions**: * Enhanced monitoring to ensure critical alerts are highlighted. * Improved visibility of alarm graphs and on-call escalation processes. 1. OpsGenie Alert Escalation Rules * **Issue**: A prior unresolved alert from October prevented escalation of new alerts. * **Actions**: * Updated the rules to ensure alerts are resolved correctly by CloudWatch signals. * Conducted a full audit of existing alert configurations to prevent recurrence. ### Key Lessons Learned * **Lifecycle Management**: Ensure sufficient ECR lifecycle retention for critical services, particularly when introducing new build artifacts. * **Alert Management**: Reduce alert fatigue by refining notification channels and escalation policies. * **Proactive Monitoring**: Implement additional safeguards to detect and address issues before they escalate to customer impact. ### Timeline * **December 24, 2024, 5:39 AM**: ECS `hooks-worker` task update/restart initiated. * **December 27, 2024, 6:29 AM**: Incident reported by Customer. * **December 27, 2024, 8:26 AM**: Deployment approved and backlog processing initiated. * **December 27, 2024, 8:56 AM**: Backlog cleared. * **December 27, 2024, 11:18 AM**: Additional impacted services redeployed. ### Closing Remarks We sincerely apologize for the inconvenience caused by this incident. We are committed to learning from this event and implementing the necessary changes to prevent similar issues in the future.
Minor December 4, 2024 Delayed execution of background tasks 2h
Started Dec 04, 2024, 11:36 PM UTC· resolved Dec 05, 2024, 01:36 AM UTC
- investigating Dec 04, 2024, 11:36 PM UTC
We are investigating an issue which is preventing background tasks from executing. Actions with new DNS zones will be delayed.
- monitoring Dec 05, 2024, 12:23 AM UTC
Issue has been resolved, we are monitoring the remaining tasks to ensure the queue is completed.
- resolved Dec 05, 2024, 01:36 AM UTC
This incident has been resolved.
Notice November 21, 2024 TLS 1.0 temporarily enabled for some sites 4h
Started Nov 21, 2024, 04:00 PM UTC· resolved Nov 21, 2024, 08:00 PM UTC
- resolved Nov 21, 2024, 04:00 PM UTC
For 4 hours, the minimum TLS was reset back to the default of 1.0. This impacted cloudcannon.com, cloudvent.net and domains added to CloudCannon before the change over to TLS 1.2 in 2021. This change was made for debugging purposes and has been reverted. Some additional bot traffic was detected during this time with no major impact on bandwidth usage. During this time 131 requests were made with TLS 1.0 and 4 requests were made with TLS 1.1. No security was compromised for legitimate human traffic.