Resolved after 4 days 2 hours, affecting Degree Map and Data Warehouse (Explore) and 1 more, on April 24, 2025.
On Tuesday April 22nd we began to notice our monitors indicating a number of workflow failures. Initially our logs pointed to DNS failures with our cloud provider. Upon troubleshooting the root cause, we identified an unhealthy node in one of our clusters that was still accepting traffic but unable to properly process data since it was in a failure state. Remediation was to manually remove the unhealthy node from the cluster at which point the issue was resolved.
Affected components
Update timeline
- identified Apr 24, 2025, 01:25 PM UTC
The issue has been identified and a fix is being developed.
- monitoring Apr 24, 2025, 07:47 PM UTC
A fix has been implemented and we are monitoring the results.
- identified Apr 24, 2025, 09:03 PM UTC
Some additional issues have been identified and we are working to resolve as quickly as possible
- identified Apr 25, 2025, 02:20 PM UTC
We are continuing to work on a fix for this issue.
- monitoring Apr 25, 2025, 02:21 PM UTC
A fix has been implemented and we are monitoring the results. Data freshness is starting to come back within the 24-48 hour threshold.
- resolved Apr 28, 2025, 04:22 PM UTC
On Tuesday April 22nd we began to notice our monitors indicating a number of workflow failures. Initially our logs pointed to DNS failures with our cloud provider. Upon troubleshooting the root cause, we identified an unhealthy node in one of our clusters that was still accepting traffic but unable to properly process data since it was in a failure state. Remediation was to manually remove the unhealthy node from the cluster at which point the issue was resolved.
Your product runs on more than Civitas Learning
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services