Resolved after 33 minutes, affecting Keeper Web Vault (US) and Keeper Admin Console (US), on June 24, 2025.
At 8:15 AM PST, our DevOps team was alerted to elevated API error rates in production. Upon investigation, we identified the following: * The production database was experiencing high table lock contention. * This was triggered by several large-scale BreachWatch data updates. * A recent backend migration had unintentionally doubled the number of parallel update operations, which exacerbated the issue. * A database failover was performed to restore service stability. The issue was fully resolved by 8:40 AM PST. The engineering team has addressed the underlying problem in the update processes and is implementing safeguards to prevent recurrence, including query optimization and improved job orchestration. We apologize for the disruption.
Affected components
Update timeline
- investigating Jun 24, 2025, 03:27 PM UTC
We are investigating vault login errors
- investigating Jun 24, 2025, 03:28 PM UTC
We are continuing to investigate this issue.
- resolved Jun 24, 2025, 04:01 PM UTC
The issue has been resolved. The outage was caused by a high volume of database locks related to a large BreachWatch data update. The engineering team is tracking down the root cause and will be addressing a preventative solution today.
- postmortem Jun 24, 2025, 08:42 PM UTC
At 8:15 AM PST, our DevOps team was alerted to elevated API error rates in production. Upon investigation, we identified the following: * The production database was experiencing high table lock contention. * This was triggered by several large-scale BreachWatch data updates. * A recent backend migration had unintentionally doubled the number of parallel update operations, which exacerbated the issue. * A database failover was performed to restore service stability. The issue was fully resolved by 8:40 AM PST. The engineering team has addressed the underlying problem in the update processes and is implementing safeguards to prevent recurrence, including query optimization and improved job orchestration. We apologize for the disruption.
Your product runs on more than Keeper
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services