KSAT - Unable to Login (DE and UK Only)

Update timeline

1. investigating Sep 18, 2026, 11:07 AM UTC

   We have received reports that users are unable to login to the DE and UK instances of KSAT. We are investigating this issue and will update this page when we have more information.

2. resolved Sep 18, 2026, 04:48 PM UTC

   This incident has been resolved.

3. postmortem Oct 08, 2026, 02:07 PM UTC

   ## **Summary** On September 17, 2026, a background job archived a large user group and ran as a single unbounded database transaction. This transaction locked a large number of user rows, causing US region logins to time out because they write to the same table. After hitting a platform execution limit, the job was terminated and automatically retried five times in 15-minute intervals, resulting in a 75-minute outage. To resolve the issue, we deployed a fix that decoupled logins from the lock. We then deployed a follow-up fix for a regression introduced by the login change, as well as for the archival job. ## **What Happened** A prior change moved the "archive all users" action from a synchronous, time-out web request into an unbounded background job. When a large user group was archived, this triggered the new background job. The job locked all affected user rows in a single transaction, performing per-user writes and a synchronous external service call before committing. Login attempts, which write to the same table, were queued behind the locked job and timed out for US users. When the job reached a 15-minute platform execution limit, it was terminated mid-transaction, the transaction rolled back, and the job was redelivered to the queue. This cycle occurred five times, resulting in a 75-minute outage. When failures were detected internally at 22:35 \(UTC\), an incident was declared, and an engineer was paged to respond. The mitigation to decouple the write action from the login flow was deployed at approximately 23:44 \(UTC\). The mitigation itself caused a regression: any user in any region who had previously mistyped a password could no longer log in because resets for failed login attempts tried to write to a read-only database. By 09:17 \(UTC\) on September 18, 2026, the fix for this regression was deployed across regions. The original US-only archive job issue recurred twice more between 13:45 and 14:05 \(UTC\) before a permanent fix was deployed at approximately 15:00 \(UTC\). The incident was fully resolved by 19:12 \(UTC\) on September 18, 2026. ## **Root Cause** An unbounded, single-transaction background job for large-group user archival held exclusive locks on many affected rows, blocking the login process's own row write. A prior change to move this operation off a web request timeout and into a background job removed the only remaining limit on its duration. The platform's 15-minute execution limit then triggered automatic kill-and-redeliver cycles, repeating the lock contention five times and resulting in a 75-minute outage window. ## **Findings and Mitigations** We found that the unbounded archive transaction locked a large number of affected user rows, blocking other table writes. This was mitigated in the short term by decoupling the login process's write dependency from the affected table, and then fixed in the long term by correcting the archive job. We also found that uncoordinated worker concurrency amplifies lock contention. This was mitigated in the short term by manually reducing worker concurrency. The long-term fix to stagger and serialize these jobs is planned. Finally, we found that the login-fix regression caused all-region login failures for users with prior failed attempts because writes were rejected by a read-only database. This was fixed via a targeted code change that was deployed and verified across all regions. ## **Preventive Measures** We’re improving our large-group archival operations by bounding them, processing them in small, committed batches, and keeping external calls outside the database transaction. We’re also adding a change-check to a specific background worker to skip writes that don't change anything. Finally, we’re adding direct alerts for database transaction age and purging delays, in addition to periodic indirect checks. ## **Customer Impact and Recovery** Login failures lasted approximately 75 minutes in the US region. Two brief, US-only recurrences occurred the next morning and were cleared manually within minutes. Some admin actions, such as user management, were also affected. Partner logins were unaffected. The secondary regression lasted around 10 hours and affected all regions for users who had previously failed a login attempt. By the afternoon of September 18, full recovery was confirmed across all regions, and the status page was marked as resolved. No data loss occurred as a result of this issue. ## **Conclusion** A single unbounded background transaction, combined with an automatic retry mechanism, caused a login outage in the US region. The specific triggering job and its retry behavior have been fixed, and login no longer depends on the contended write path. A same-incident regression and two minor recurrences were each resolved within hours. Remaining preventive work, including worker coordination and database transaction monitoring, is tracked above.