Secret Server EU: Errors launching sessions and delayed background tasks

Update timeline

1. investigating Oct 01, 2026, 09:22 PM UTC

   We are currently investigating an ongoing issue in the EU region. We'll provide an update soon.

2. monitoring Oct 01, 2026, 10:09 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Oct 02, 2026, 12:25 AM UTC

   This incident has been resolved.

4. postmortem Oct 14, 2026, 12:28 AM UTC

   ## Incident Overview On October 1, 2026, customers in the EU region had problems using Secret Server Cloud \(SSC\). Users could not launch sessions from secrets, and pages for viewing secret details and audit history loaded slowly or failed. Scheduled background tasks such as heartbeats, password changes, discovery and notifications were delayed, and some Distributed Engines could not connect. * Start Time: October 1, 2026, 08:38 PM UTC * End Time: October 2, 2026, 12:30 AM UTC ## Root Cause and Remediation SSC uses a messaging service to pass work between its components. Examples include recording secret view activity for auditing, running scheduled tasks and preparing session launches. On September 30, 2026, the SSC service that processes these messages in the EU region began handling work at a much lower rate than normal. New activity kept arriving at the normal rate, so unprocessed messages built up in the queue. When the queue reached its capacity, the extra messages were moved to a holding area for undeliverable messages. Over the following hours this holding area kept growing until the messaging service hit its storage limit. Our engineering team increased the capacity of the messaging service, scaled up and restarted the SSC EU services, and cleared the backlog of undeliverable messages. Once the backlog drained, session launches, page loads and scheduled tasks returned to normal by October 2, 2026, 12:30 AM UTC. Delayed background tasks, including heartbeats, password changes and notifications, were processed automatically after service was restored. Customers did not need to take any action. Because the services were restored immediately to end customer impact, we did not capture the diagnostic data needed to determine why their throughput dropped, so the underlying trigger remains unknown. The queue growth should have been caught earlier. During a recent migration of our alerting tooling, a configuration change sent certain cloud metric alerts, including the messaging queue size alert, to the wrong destination. That alert triggered on the morning of October 1 but never notified our on-call team. Our automated remediation for other alert types was unaffected, which masked the gap. ## Preventative Actions 1. We corrected the alert routing on October 2, 2026, so these alerts now reach our on-call team, and we are adding canary and heartbeat alerts that notify us if alert delivery from our monitoring tools stops or drops unexpectedly. 2. We are strengthening how we change alerting, by testing each alert type end to end before and after any production deployment. 3. We are monitoring for a recurrence of the slowdown and will capture the diagnostic data needed to determine its cause if it happens again. 4. We are updating our deployment cycle with a safeguard that ensures the background services are refreshed even when a release is delayed, and we are adding this step to our release checklist. We apologize for the disruption this caused and appreciate your patience while we resolved it.