Resolved after 31 minutes, affecting Response Time, on September 30, 2026.
**Issue Summary:** On September 30, 2026, the US SL3 Production environment experienced delays in processing queued operations, resulting in an unusually high volume of events. Affected operations were unable to process at the expected rate, causing events to accumulate in the queue and delaying completion of activities. **Root Cause:** The issue was caused by database deadlocks occurring during concurrent data merge operations. When these conflicts occurred, the application attempted to retry the affected operations without sufficiently reducing the batch size of learning objects being processed. This resulted in repeated database conflicts, preventing queued events from being processed efficiently and causing an accumulation of events. **Corrective Action:** Cornerstone implemented a fix to improve the application's retry logic and database deadlock handling. The update addressed repeated processing conflicts by adjusting how failed operations are retried, allowing data merge operations to be processed more efficiently. **Preventive Measures:** To reduce the risk of a similar issue, Cornerstone has taken the following actions: * Enhanced retry handling to reduce batch sizes when processing conflicts occur, minimizing the likelihood of repeated database deadlocks. * Improved error handling to ensure database deadlocks are properly identified and managed, preventing repeated attempts to process the same failed operations.
Affected components
Update timeline
- investigating Sep 30, 2026, 04:08 PM UTC
US SL3 swim lane is experiencing delays in processing background or queued tasks. During this time, background tasks, including email processing, may take longer than usual to complete. Resolving this issue is our top priority, and we are working to fix it as quickly as possible. Please check back periodically for updates, which will be posted as they become available.
- resolved Sep 30, 2026, 04:39 PM UTC
CSOD Technology Team observed delays in processing background or queued tasks on US SL3 swim lane including emails. The problem began on 09/30/2026 06:27 AM - PDT and was resolved on 09/30/2026 09:20 AM - PDT. During this time, background tasks may have taken longer than normal to complete.
- postmortem Oct 22, 2026, 08:53 PM UTC
**Issue Summary:** On September 30, 2026, the US SL3 Production environment experienced delays in processing queued operations, resulting in an unusually high volume of events. Affected operations were unable to process at the expected rate, causing events to accumulate in the queue and delaying completion of activities. **Root Cause:** The issue was caused by database deadlocks occurring during concurrent data merge operations. When these conflicts occurred, the application attempted to retry the affected operations without sufficiently reducing the batch size of learning objects being processed. This resulted in repeated database conflicts, preventing queued events from being processed efficiently and causing an accumulation of events. **Corrective Action:** Cornerstone implemented a fix to improve the application's retry logic and database deadlock handling. The update addressed repeated processing conflicts by adjusting how failed operations are retried, allowing data merge operations to be processed more efficiently. **Preventive Measures:** To reduce the risk of a similar issue, Cornerstone has taken the following actions: * Enhanced retry handling to reduce batch sizes when processing conflicts occur, minimizing the likelihood of repeated database deadlocks. * Improved error handling to ensure database deadlocks are properly identified and managed, preventing repeated attempts to process the same failed operations.
Your product runs on more than Cornerstone
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services