Resolved on January 25, 2025.
**What Happened** On Saturday nights, we run a full reindex operation on each database before the Full backups run against our databases. During the reindex operation, our application servers started having issues connecting to the databases. **Root Cause** A recent change in the application stack caused a significantly higher rate of log entries to a system Log table. Due to the increased size of this table, the reindex process locked that table for a significant amount of time. In turn, this caused issues for all OC application servers connecting to the databases. **Corrective Actions** * A code change has been released to production to fix the uncontrolled growth of the log table. * Enhance monitors that will alert when unexpected growth conditions happen on key system Log tables.
Update timeline
- resolved Jan 26, 2025, 03:23 AM UTC
Some sites were showing a server error when trying to access
- postmortem Jan 30, 2025, 02:01 PM UTC
**What Happened** On Saturday nights, we run a full reindex operation on each database before the Full backups run against our databases. During the reindex operation, our application servers started having issues connecting to the databases. **Root Cause** A recent change in the application stack caused a significantly higher rate of log entries to a system Log table. Due to the increased size of this table, the reindex process locked that table for a significant amount of time. In turn, this caused issues for all OC application servers connecting to the databases. **Corrective Actions** * A code change has been released to production to fix the uncontrolled growth of the log table. * Enhance monitors that will alert when unexpected growth conditions happen on key system Log tables.
Your product runs on more than Higher Logic
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services