Socure Outage History

Started Oct 27, 2026, 04:00 PM UTC· resolved Oct 27, 2026, 04:00 PM UTC

1. resolved Oct 27, 2026, 04:00 PM UTC

   On October 27, 2026, some customers were unable to receive RiskOS Sandbox webhooks. The issue was caused by invalid webhook configurations that caused the Sandbox notification sender to stop processing events. The malformed configurations have since been corrected. All processing has recovered and webhook delivery has successfully resumed

Read the full incident report →

Started Oct 23, 2026, 09:00 PM UTC· resolved Oct 23, 2026, 09:12 PM UTC

1. monitoring Oct 23, 2026, 09:00 PM UTC

   SSA is currently experiencing a service-impacting event affecting multiple applications and services, including the Electronic Consent Based Social Security Number Verification (eCBSV) service. A small number of transactions are affected as the disruption was first noted at approximately 4:47 PM (ET) on October 23, 2026. We will share another update once we receive notice that service has been restored, and eCBSV is operating normally again.

2. resolved Oct 23, 2026, 09:12 PM UTC

   At approximately 4:56 (ET), service has been restored and eCBSV is operating normally at this time. We apologize for any inconveniences this may have caused.

Read the full incident report →

Started Oct 19, 2026, 07:04 PM UTC· resolved Oct 20, 2026, 01:57 AM UTC

1. investigating Oct 19, 2026, 07:04 PM UTC

   We are currently investigating an issue that is causing intermittent latency with displaying and downloading DocV Images for a small percentage of completed transactions. There is no impact to DocV nor evaluations at this time but images may take additional time to appear.

2. monitoring Oct 19, 2026, 09:15 PM UTC

   Our team has taken steps to mitigate the latency experienced for DocV image display and downloads. There continues to be no impact to DocV evaluations at this time and the team will continue to monitor the situation. We will provide another update once all impact has been resolved.

3. resolved Oct 20, 2026, 01:57 AM UTC

   This issue has been resolved, and we’re monitoring performance to ensure the services remain stable. We apologize for any inconvenience this may have caused.

Read the full incident report →

Started Oct 18, 2026, 06:50 PM UTC· resolved Oct 18, 2026, 07:10 PM UTC

1. resolved Oct 18, 2026, 07:33 PM UTC

   SSA experienced a service-impacting event that affected multiple applications and services, including the Electronic Consent Based Social Security Number Verification (eCBSV) service. A small number of transactions were affected as the disruption began at approximately 2:50PM (ET) on October 18, 2026, and was resolved at approximately 3:10PM (ET). Service has been restored, and eCBSV is operating normally at this time.

Read the full incident report →

Started Oct 16, 2026, 08:20 PM UTC· resolved Oct 17, 2026, 02:10 AM UTC

1. investigating Oct 16, 2026, 08:20 PM UTC

   Starting at 3:00 EDT, some customers may experience intermittent errors or failed requests when using the RiskOS Evaluate APIs. We’ve identified a database issue and are actively working to resolve it. We apologize for the disruption and appreciate your patience while we investigate.

2. monitoring Oct 16, 2026, 09:18 PM UTC

   We’ve taken an interim measure, and the elevated error rate has subsided. We haven’t observed related errors in the past 15 minutes. We’re moving the incident to monitoring while we continue work on a full resolution. We’ll share another update when the situation changes.

3. monitoring Oct 16, 2026, 10:59 PM UTC

   We appreciate your patience as our team is still monitoring the situation closely. Error rates have subsided as we continue to work towards a full resolution.

4. resolved Oct 17, 2026, 02:10 AM UTC

   The incident has been resolved, and we are no longer seeing errors. Services are operating normally, and we’ll continue to monitor performance. We appreciate your patience while we worked through this.

Read the full incident report →

Started Oct 15, 2026, 10:39 PM UTC· resolved Oct 15, 2026, 11:08 PM UTC

1. investigating Oct 15, 2026, 10:39 PM UTC

   We are currently experiencing an active incident that is impacting service availability. We are currently investigating the issue and will provide updates as soon as we have additional information.

2. monitoring Oct 15, 2026, 11:04 PM UTC

   Socure RiskOS and ID+ services have been restored. An error in a rollout caused a temporary spike in error rates from our services between 18:20 and 18:56 Eastern time. The error has been corrected and services are stable now. We understand this outage is unacceptable and a full RCA will be available as soon as full analysis of the issue is complete.

3. monitoring Oct 15, 2026, 11:04 PM UTC

   Socure RiskOS and ID+ services have been restored. An error in a rollout caused a temporary spike in error rates from our services between 18:12 and 18:56 Eastern time. The error has been corrected and services are stable now. We understand this outage is unacceptable and a full RCA will be available as soon as full analysis of the issue is complete.

4. resolved Oct 15, 2026, 11:08 PM UTC

   Socure RiskOS and ID+ services have been restored. An error in a rollout caused a temporary spike in error rates from our services between 18:12 and 18:56 Eastern time. The error has been corrected and services are stable now. We understand this outage is unacceptable and a full RCA will be available as soon as full analysis of the issue is complete.

5. postmortem Oct 17, 2026, 01:30 AM UTC

   ## **Root Cause Analysis** **Incident:** Socure APIs Outage **Date Range:** 15-Oct-2026 **Impact:** Interruption in service for Socure APIs **Status:** Resolved ### **1. Summary** On 15-Oct-2026, between 18:12 and 19:00 Eastern time Socure experienced an interruption to its RiskOS and ID\+ APIs, causing errors and timeouts to users. The first phase ran from 18:12 to 18:20, and was a major interruption for virtually every service. The second phase of the outage ran from 18:20 to 19:00, and was a partial outage for services that rely on KYC data, where customers saw elevated error rates and latencies for some services, but traffic was still being served. ### **2. Timeline** **Date / Time \(ET\)** | Event **18:12** | A routine cleanup change is merged, moving a source repo from one code repository to another. A bug in our review tool hides the scope of the change. **18:13** | ArgoCD restarts most services in a major Kubernetes cluster all at once. **18:20** | All service pods are running. Partial API service restored, although KYC services are still seeing an elevated error rate. **18:21** | The flood of startup traffic overwhelms our KYC Search service. Error rates spike. **18:42** | SRE diagnoses the cascading failure impacting KYC Search and blocks traffic to it so it can come up. **18:56** | The KYC Search service is fully up, and SRE returns traffic to it. Downstream services stabilize. **19:00** | All traffic running as normal, incident declared stable. ### **3. Root Cause** * **Primary Root Cause:** As a part of an ongoing migration, an on-call SRE made a change to ArgoCD, the system we use to manage the kubernetes clusters Socure runs its services on. The change was scoped to a single cluster, was reviewed carefully before pushing, caused no issues in other environments it was tested in, and should have had no visible impact. Unfortunately, the change updated a legacy setting that did not appear in the change review. That legacy setting governed what services ArgoCD managed, and removed many live services from ArgoCD. In consequence, ArgoCD removed those services from production briefly, before the setting was corrected. * **Contributing Factors:** The KYC Search service is a backend piece of Socure infrastructure that provides services to several of our internal products. When that service restarted from scratch at the same time as all of the clients of the service did, a small number of pods in the service crashed under load. This increased the load on the remaining pods, some of which crashed in turn. This cascading failure prevented the service from coming back on its own the way the rest of Socure’s infrastructure did. Diagnosing this separate failure mode took additional time, and remediating it fixed the remaining errors after the initial incident. ### **4. Resolution** The ArgoCD change was reverted immediately, causing services to come back to their prior state. The KYC Search service required manual effort to block traffic from reaching it until a majority of pods in the service could come up. After the service was healthy, we could restore traffic to it and resolve the overall incident. ### **5. Corrective and Preventive Actions** ### **6. Lessons Learned** * Socure’s tooling for viewing the changes it makes to production did not work correctly. We are reviewing these tools to ensure they operate correctly under all use cases and fix any cases like this where the full scope of a change is not clear. * Augment the existing change review process with agentic support. While in this case, the specific change was not visible in the plan, having an extra reviewer that is supported by an extensive training model will help ensure issues are caught before submission. * Even changes that appear to have no impact, when they are made to the governance system for our production jobs, can be risky, and should be shifted further out of US business hours. * Our KYC Search service is vulnerable to a “thundering-herd” problem when it receives full traffic flow when it is not fully online. We need to strengthen our circuit breakers to prevent this issue recurring. * We run too many of our core API services in a single kubernetes cluster. Even though that cluster is distributed across multiple availability zones, the shared control plane creates too high a blast radius for changes to that cluster. ### **7. Next Steps & Ongoing Commitment** Socure understands our customers rely on us as a key provider in critical business flows, and that this service interruption is not acceptable. We apologize for the impact this issue caused, and commit to ensuring it does not recur. To that end, we will deliver details on our plans to re-architect our application layout by November, as well as improve the resilience of the KYC Search service in Q1 of next year.

Read the full incident report →

Started Oct 15, 2026, 10:12 PM UTC· resolved Oct 15, 2026, 11:00 PM UTC

1. resolved Oct 16, 2026, 02:23 AM UTC

   Socure RiskOS and ID+ services have been restored. An error in a rollout caused a temporary spike in error rates from our services between 18:12 and 18:56 Eastern time. The error has been corrected and services are stable now. We understand this outage is unacceptable and a full RCA will be available as soon as full analysis of the issue is complete.

2. postmortem Oct 17, 2026, 01:42 AM UTC

   ## **Root Cause Analysis** **Incident:** Socure APIs Outage **Date Range:** 15-Oct-2026 **Impact:** Interruption in service for Socure APIs **Status:** Resolved ### **1. Summary** On 15-Oct-2026, between 18:12 and 19:00 Eastern time Socure experienced an interruption to its RiskOS and ID\+ APIs, causing errors and timeouts to users. The first phase ran from 18:12 to 18:20, and was a major interruption for virtually every service. The second phase of the outage ran from 18:20 to 19:00, and was a partial outage for services that rely on KYC data, where customers saw elevated error rates and latencies for some services, but traffic was still being served. ### **2. Timeline** **Date / Time \(ET\)** | Event **18:12** | A routine cleanup change is merged, moving a source repo from one code repository to another. A bug in our review tool hides the scope of the change. **18:13** | ArgoCD restarts most services in a major Kubernetes cluster all at once. **18:20** | All service pods are running. Partial API service restored, although KYC services are still seeing an elevated error rate. **18:21** | The flood of startup traffic overwhelms our KYC Search service. Error rates spike. **18:42** | SRE diagnoses the cascading failure impacting KYC Search and blocks traffic to it so it can come up. **18:56** | The KYC Search service is fully up, and SRE returns traffic to it. Downstream services stabilize. **19:00** | All traffic running as normal, incident declared stable. ### **3. Root Cause** * **Primary Root Cause:** As a part of an ongoing migration, an on-call SRE made a change to ArgoCD, the system we use to manage the kubernetes clusters Socure runs its services on. The change was scoped to a single cluster, was reviewed carefully before pushing, caused no issues in other environments it was tested in, and should have had no visible impact. Unfortunately, the change updated a legacy setting that did not appear in the change review. That legacy setting governed what services ArgoCD managed, and removed many live services from ArgoCD. In consequence, ArgoCD removed those services from production briefly, before the setting was corrected. * **Contributing Factors:** The KYC Search service is a backend piece of Socure infrastructure that provides services to several of our internal products. When that service restarted from scratch at the same time as all of the clients of the service did, a small number of pods in the service crashed under load. This increased the load on the remaining pods, some of which crashed in turn. This cascading failure prevented the service from coming back on its own the way the rest of Socure’s infrastructure did. Diagnosing this separate failure mode took additional time, and remediating it fixed the remaining errors after the initial incident. ### **4. Resolution** The ArgoCD change was reverted immediately, causing services to come back to their prior state. The KYC Search service required manual effort to block traffic from reaching it until a majority of pods in the service could come up. After the service was healthy, we could restore traffic to it and resolve the overall incident. ### **5. Corrective and Preventive Actions** ### **6. Lessons Learned** * Socure’s tooling for viewing the changes it makes to production did not work correctly. We are reviewing these tools to ensure they operate correctly under all use cases and fix any cases like this where the full scope of a change is not clear. * Augment the existing change review process with agentic support. While in this case, the specific change was not visible in the plan, having an extra reviewer that is supported by an extensive training model will help ensure issues are caught before submission. * Even changes that appear to have no impact, when they are made to the governance system for our production jobs, can be risky, and should be shifted further out of US business hours. * Our KYC Search service is vulnerable to a “thundering-herd” problem when it receives full traffic flow when it is not fully online. We need to strengthen our circuit breakers to prevent this issue recurring. * We run too many of our core API services in a single kubernetes cluster. Even though that cluster is distributed across multiple availability zones, the shared control plane creates too high a blast radius for changes to that cluster. ### **7. Next Steps & Ongoing Commitment** Socure understands our customers rely on us as a key provider in critical business flows, and that this service interruption is not acceptable. We apologize for the impact this issue caused, and commit to ensuring it does not recur. To that end, we will deliver details on our plans to re-architect our application layout by November, as well as improve the resilience of the KYC Search service in Q1 of next year.

Read the full incident report →

Started Oct 15, 2026, 02:52 PM UTC· resolved Oct 15, 2026, 03:59 PM UTC

1. identified Oct 15, 2026, 02:52 PM UTC

   We are currently experiencing an unplanned outage affecting the eCBSV module due to SSA responses, resulting in 5xx errors. Socure has contacted the vendor to investigate the issue and determine when service is expected to be restored.

2. monitoring Oct 15, 2026, 02:55 PM UTC

   Error rates have subsided over the last few minutes. We are monitoring these rates, and pending our vendor response.

3. resolved Oct 15, 2026, 03:59 PM UTC

   SSA experienced a service-impacting event that affected multiple applications and services, including the Electronic Consent Based Social Security Number Verification (eCBSV) service. The disruption began at approximately 10:19AM (ET) on October 15, 2026, and was resolved at approximately 10:41AM (ET). Service has been restored, and eCBSV is operating normally at this time.

Read the full incident report →

Started Oct 09, 2026, 11:25 PM UTC· resolved Oct 09, 2026, 11:25 PM UTC

1. resolved Oct 09, 2026, 11:25 PM UTC

   We experienced elevated transaction failures affecting Watchlist Plus and Watchlist Premier. The issue has been resolved, and we’re monitoring performance to ensure the services remain stable. We apologize for any inconvenience this may have caused.

Read the full incident report →

Started Sep 22, 2026, 12:26 PM UTC· resolved Sep 22, 2026, 02:09 PM UTC

1. identified Sep 22, 2026, 12:26 PM UTC

   We are investigating an issue causing delays in case updates being reflected in the RiskOS Case Management UI. Our Engineering team is actively working to identify the root cause and restore normal update behavior. We will provide further updates as more information becomes available. This is not impacting API evaluation requests.

2. monitoring Sep 22, 2026, 01:22 PM UTC

   We have identified the root cause of the issue affecting updates in the RiskOS Case Management UI. This issue is limited to a subset of cases and specifically affects the Case Search functionality, which may temporarily show outdated case states. Please note that case status data displayed within the Case List and individual Case Details pages is accurate and reflects the latest state. Our Engineering team has deployed a fix for all newly updated cases and is actively working to re-sync the remaining affected cases to ensure data consistency across search. We will provide another update once normal behavior is fully restored.

3. resolved Sep 22, 2026, 02:09 PM UTC

   The issue affecting case status updates has been resolved. All the new cases are being processed correctly. We sincerely apologize for any inconvenience this may have caused and appreciate your patience while our team worked to resolve it.

Read the full incident report →

Started Sep 21, 2026, 10:28 AM UTC· resolved Sep 21, 2026, 10:57 AM UTC

1. resolved Sep 21, 2026, 11:57 AM UTC

   Between 6:28 AM and 6:57 AM ET on September 21, an issue during an operational activity caused some webhook notifications associated with older evaluations from the past month to be delivered again. Current evaluations and webhook processing were not impacted. The issue has been resolved. We apologize for any confusion or additional noise this may have caused.

Read the full incident report →

Started Sep 16, 2026, 02:36 PM UTC· resolved Sep 16, 2026, 02:49 PM UTC

1. investigating Sep 16, 2026, 02:36 PM UTC

   We are investigating delays affecting batch processing on the dashboard. Customers may experience delayed processing while we review the issue.

2. resolved Sep 16, 2026, 02:49 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 16, 2026, 09:39 AM UTC· resolved Sep 16, 2026, 10:12 AM UTC

1. identified Sep 16, 2026, 09:39 AM UTC

   We are observing increased latency affecting some customers using our Fraud and Synthetic products. Our team is actively investigating the issue. API requests are completing successfully, and we are not observing any API failures at this time. We will provide further updates as more information becomes available.

2. monitoring Sep 16, 2026, 10:07 AM UTC

   We’re seeing an improvement in latency affecting some customers using our Fraud and Synthetic products. We’ll continue to monitor the situation closely.

3. resolved Sep 16, 2026, 10:12 AM UTC

   The issue impacting Fraud and Synthetic services has been resolved, and service has returned to normal operation. Customers should no longer experience increased latency or delays when accessing Fraud and Synthetic services

Read the full incident report →

Started Sep 10, 2026, 01:30 PM UTC· resolved Sep 10, 2026, 07:00 PM UTC

1. resolved Sep 10, 2026, 07:39 PM UTC

   Between 8:30 AM and 2:00 PM EST, we observed an increase in 5xx error rates, which resulted in intermittent errors or failed requests for some customers. Overall impact was minimal, less than 0.01%. The error rate has since returned to normal. This issue is fully resolved at this time. We apologize for any disruption and appreciate your patience.

Read the full incident report →

Started Sep 09, 2026, 11:00 AM UTC· resolved Sep 09, 2026, 05:13 PM UTC

1. resolved Sep 09, 2026, 11:37 AM UTC

   Between 7:00 AM and 7:20 AM EST, we observed an increase in 5xx error rates, which may have resulted in intermittent errors or failed requests for some customers. The error rate has since returned to normal, and we are investigating the root cause to ensure we have fully resolved the issue. We apologize for any disruption and appreciate your patience.

2. postmortem Sep 15, 2026, 06:15 PM UTC

   **Root Cause Analysis** **Incident:** Intermittent elevated 5xx errors across services **Date Range:** September 9, 2026, 7:00 AM – 9:20 AM EDT \(approximately 2 hours 20 minutes of intermittent impact\) **Impact:** Some requests submitted to the EmailAuthScore API, and related services, intermittently returned error responses. Impact was intermittent rather than continuous and was limited to defined windows described in the timeline below. **Status:** Resolved ### **1. Summary** On September 9, 2026, a planned increase in backend processing capacity led to elevated connection load on an internal infrastructure component used to support downstream processing for several services. This component reached an internal capacity threshold, which introduced processing delays and resulted in intermittent error responses on the EmailAuthScore API and related services for a subset of requests. The issue was identified through a combination of internal monitoring and reports during the impact window. Socure engineering correlated the errors with the recent capacity change and reverted it, which stopped the initial increase in errors. A second, related increase in errors then occurred when a separate capacity change, made earlier for a different internal service, placed additional load on the same shared component; engineering applied further mitigations, after which all affected services returned to normal, stable operation. Section 5 outlines the corrective and preventive actions Socure is committing to, with priority on improving visibility into infrastructure capacity and preventing delays in any single internal component from affecting customer-facing request processing. ### **2. Timeline** | **Date / Time \(EDT\)** | **Event** | | --- | --- | | **September 9, 2026 07:00 AM** | Following a planned increase in backend processing capacity, the EmailAuthScore API began intermittently returning error responses. | | **September 9, 2026 07:20 AM** | The recent capacity change was reverted; error rates returned to normal. | | **September 9, 2026 08:45 AM** | A separate capacity change made earlier for a different internal service placed additional load on the same shared internal component, and the EmailAuthScore API again began intermittently returning error responses. | | **September 9, 2026 09:20 AM** | Additional mitigations were applied; error rates returned to normal and remained stable. | | **September 9, 2026 01:13 PM** | As a precaution, Socure temporarily rerouted a portion of internal processing away from the affected internal component to safely complete the remaining capacity increase. No further errors were observed after this change. | ### **3. Root Cause** * **Primary Root Cause:** As part of planned work to increase system capacity ahead of anticipated demand, Socure increased the processing capacity of backend services. This change increased the number of connections to an internal messaging component used to support downstream processing for these services. That component reached an internal connection capacity limit, which resulted in connection and processing delays. These delays introduced additional latency and, in some cases, backpressure into request processing, which contributed to intermittent elevated error rates on the EmailAuthScore API and related services for a subset of requests during the impact windows described above.. * **Contributing Factors:** * A sudden increase in infrastructure capacity demand exceeded the capacity that was immediately not available within the internal messaging component. ### **4. Resolution** Upon identifying that the elevated error rates were linked to the recent capacity change, Socure engineering reverted the change, which stopped the initial increase in errors. A second, related increase in errors then occurred when a separate capacity change, made earlier for a different internal service, placed additional load on the same shared component. Engineering applied additional mitigations, including temporarily rerouting a portion of internal processing away from the affected component for the impacted services. Following these actions, all affected services returned to normal, stable operation, which Socure confirmed through continued monitoring. ‌ ### **5. Corrective and Preventive Actions** | **Action** | **Description** | **ETA / Status** | | --- | --- | --- | | Optimize infrastructure capacity utilization | Adjusted backend service capacity and connection usage to ensure processing remains within supported infrastructure limits and to reduce the risk of similar capacity-related issues. | Done | | Increase messaging infrastructure capacity | Increased the capacity of the underlying messaging infrastructure to provide additional headroom and support future increases in traffic volume. | Done | | Enhance capacity monitoring and alerting | Add proactive monitoring and alerts to identify potential infrastructure capacity constraints before they affect service performance. | In progress. ETA 30 Sep 2026. | ### **6. Lessons Learned** * Proactive monitoring of infrastructure health needs to be in place ahead of capacity changes, not only application-level monitoring. ### **7. Next Steps & Ongoing Commitment** We are committed to: * Completing the corrective and preventive actions listed in Section 5. * Improving visibility into the health and capacity of critical internal infrastructure ahead of future scaling activity. * Providing a follow-up update once all corrective actions are complete.

Read the full incident report →

Started Sep 08, 2026, 02:54 PM UTC· resolved Sep 08, 2026, 03:41 PM UTC

1. investigating Sep 08, 2026, 02:54 PM UTC

   We are currently investigating an issue that created an impact on DocV and Watchlist monitoring services where errors were observed today September 8th, from 09:05 AM EST to 09:30 AM EST. System has recovered, we are working on determining a root cause at this time.

2. resolved Sep 08, 2026, 03:41 PM UTC

   The issue impacting DocV and Watchlist monitoring services on September 8 has been resolved. No customer impact has been observed since 10:35 AM EST. We apologize for any disruption this may have caused.

3. postmortem Sep 17, 2026, 07:47 PM UTC

   #### **Root Cause Analysis** **Incident:** Service Interruption Affecting Account Data Access **Date Range:** September 8, 2026 \(10:04 AM to 10:31 AM EDT\) **Impact:** DocV and Watchlist Monitors **Status:** Resolved ### 1. Summary An unexpected depletion of backend database capacity temporarily prevented our platform from retrieving account information for a subset of client requests. During the incident window \(10:04 AM to 10:31 AM EDT of 8-Sep-2026\), some of the client requests experienced elevated error rates when attempting to query account details. Following a routine maintenance update, concurrent connections from backup environments exceeded normal thresholds, saturating available database capacity. Engineering teams safely disconnected the redundant environments and refreshed active connections, restoring full system responsiveness and stability. ### 2. Timeline | **Date / Time \(ET\)** | **Event** | | --- | --- | | First error | 10:04 AM EST, 8 Sep 2026 | | Incident reported | 10:17 AM EST, 8 Sep 2026 | | Redundant environment connection pool isolated | 10:30 AM EST, 8 Sep 2026 | | Incident resolved | 10:31 AM EST, 8 Sep 2026 | ### 3. Root Cause * **Primary Root Cause:** During a scheduled system update, elevated connection retention between deployment environments led to database connection pool saturation. Under peak traffic, this caused processing delays and request timeouts for dependent platform services. * **Contributing Factors:** * Rigid connection timeout parameters delayed the timely release of idle database allocations by passive environment. * Inadequate database node headroom hindered seamless processing of extra connections generated by the standby environment. * **Non-contributing Factors:** * None ### 4. Resolution Full service integrity was restored through the following immediate actions: 1. Isolating non-active environment connections to immediately free database pool capacity. 2. Performing a rolling reset of active service nodes to re-establish clear database connection baselines. ‌ ### 5. Corrective and Preventive Actions | **Action** | **Description** | **ETA / Status** | | --- | --- | --- | | Improvements to data access patterns | Improve caching strategy and data access patterns in internal services that rely on accounts data. | 15 Oct 2026 | | Automated Connection Management | Enhance connection lifecycle management and automated cleanup routines for inactive background tasks. | 15 Oct 2026 | ### 6. Lessons Learned * **Enhancing Deployment Isolation:** We are updating our deployment procedures to ensure complete separation of resources across active and passive environments, eliminating cross-environment resource contention during upgrades. * **Optimizing Database Resource Allocation:** To ensure peak performance, we are rebalancing our workload and increasing overall system capacity and efficiency. ### 7. Next Steps & Ongoing Commitment We are committed to: * Resolve the identified root causes by the timelines shared * Implement proactive safety guards and monitoring thresholds while permanent application and infrastructure enhancements are finalized.

Read the full incident report →

Started Sep 02, 2026, 06:31 PM UTC· resolved Sep 03, 2026, 03:18 PM UTC

1. identified Sep 02, 2026, 06:31 PM UTC

   Our Vendor, SSA, has reported that there is currently an elevated latency issue affecting eCBSV module. As a result, you may experience elevated response times. Our team is actively working with them to resolve the issue as quickly as possible. We apologize for any inconvenience this may cause and appreciate your patience and understanding.

2. resolved Sep 03, 2026, 03:18 PM UTC

   SSA resolved the elevated latency affecting the eCBSV module as of 2:00 AM ET. Response times have returned to normal. Thank you for your patience and understanding.

Read the full incident report →

Started Aug 10, 2026, 03:40 PM UTC· resolved Aug 10, 2026, 04:02 PM UTC

1. investigating Aug 10, 2026, 03:40 PM UTC

   We are investigating a slight degradation on the Watchlist product due to latency. Affecting both ID+ and RiskOS. The team is actively investigating this issue at this time.

2. identified Aug 10, 2026, 03:40 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Aug 10, 2026, 03:52 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Aug 10, 2026, 04:02 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 04, 2026, 05:21 AM UTC· resolved Aug 04, 2026, 06:15 AM UTC

1. resolved Aug 04, 2026, 05:00 AM UTC

   Between 1:21 AM and 2:15 AM EDT on August 4, 2026, a subset of DocV webhook delivery notifications experienced elevated 401 response rates. The issue is fully resolved and webhook delivery is now operating normally. A full Root Cause Analysis (RCA) will be shared once the investigation concludes.

Read the full incident report →

Started Aug 03, 2026, 04:54 PM UTC· resolved Aug 03, 2026, 05:32 PM UTC

1. investigating Aug 03, 2026, 04:54 PM UTC

   We are currently investigating reports of increased latency for submitted transactions, which may result in processing delays. Our team is actively looking into the root cause and will provide updates as more information becomes available. We apologize for any inconvenience and appreciate your patience.

2. identified Aug 03, 2026, 05:08 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Aug 03, 2026, 05:20 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Aug 03, 2026, 05:32 PM UTC

   A fix has been implemented and confirmed effective. All systems are operating normally. Thank you for your patience while we worked through this.

Read the full incident report →

Started Jul 17, 2026, 03:06 PM UTC· resolved Jul 17, 2026, 03:47 PM UTC

1. investigating Jul 17, 2026, 03:06 PM UTC

   We are currently investigating elevated 400 (Invalid Request) errors affecting requests to enable Watchlist Monitoring subscriptions. This issue has been ongoing since approximately 6:20 AM ET. We will provide updates as more information becomes available.

2. identified Jul 17, 2026, 03:26 PM UTC

   We have identified the root cause of the elevated errors affecting requests. Our team is currently working on a fix to address the issue. We will provide a further update once the fix has been applied.

3. monitoring Jul 17, 2026, 03:41 PM UTC

   A release has been completed to address the elevated errors affecting Watchlist Monitoring subscription enablement. We have not observed any further errors in the last several minutes and are continuing to monitor to confirm full resolution.

4. resolved Jul 17, 2026, 03:47 PM UTC

   We have confirmed that Watchlist Monitoring subscription enablement has returned to normal operation, with no further errors observed. This issue is now resolved. Thank you for your patience while we worked through this.

Read the full incident report →

Started Jul 14, 2026, 01:23 PM UTC· resolved Jul 14, 2026, 01:31 PM UTC

1. resolved Jul 14, 2026, 02:17 PM UTC

   On July 14, we experienced an issue that resulted in elevated 5XX errors affecting Watchlist Screening requests approximately between 9:23 AM and 9:31 AM EDT. This issue impacted Watchlist Plus and Watchlist Premier services only. We apologize for the inconvenience and appreciate your patience.

Read the full incident report →

Started Jun 26, 2026, 08:06 PM UTC· resolved Jun 26, 2026, 08:26 PM UTC

1. investigating Jun 26, 2026, 08:06 PM UTC

   We are currently experiencing 5xx errors affecting multiple modules. Our team is actively investigating the issue and will provide updates as more information becomes available.

2. resolved Jun 26, 2026, 08:26 PM UTC

   Socure services have now fully recovered and are operating normally. Our engineering and operations teams will continue to work to conduct a comprehensive root cause analysis (RCA) of the incident. We will share the RCA publicly within the next few days once this review is complete.

Read the full incident report →

Started Jun 25, 2026, 06:06 PM UTC· resolved Jun 26, 2026, 10:03 AM UTC

1. resolved Jun 26, 2026, 04:30 PM UTC

   Between Thursday, June 25, 2026, at 2:06 PM EDT and Friday, June 26, 2026, at 6:03 AM EDT, customers subscribed to webhook events experienced delayed webhook deliveries due to an issue affecting our webhook processing pipeline. During this period, approximately half of the webhook deliveries were delayed. No webhook events were lost. The processing backlog did not surface within the webhook workers until 5:06 AM EDT on June 26. Once it became visible, the queued webhook events were processed rapidly without manual intervention, and webhook delivery latency returned to normal by 6:03 AM EDT. We are continuing to investigate why a subset of webhook events became blocked from processing and why the backlog later surfaced without intervention. We will publish a detailed root cause analysis (RCA), including corrective actions, once our investigation is complete.

2. postmortem Jul 09, 2026, 12:31 PM UTC

   ## **Root Cause Analysis** **Incident:** RiskOS Delayed Webhook Deliveries **Date Range:** June 25, 2026 2:06 PM ET to June 26, 2026 6:03 AM ET **Impact:** Customers subscribed to webhook events experienced delayed webhook deliveries during the incident window. Approximately half of webhook events were delayed; no webhook events were lost. The webhook processing pipeline auto recovered without manual intervention, and no customer action was required. **Status:** Resolved **1.Summary** Between approximately 2:06 PM ET on June 25, 2026 and 6:03 AM ET on June 26, 2026, customers subscribed to RiskOS webhook events experienced delayed webhook deliveries due to an unexpected degradation in the webhook processing pipeline. During the impact window, approximately half of webhook events were delayed. No webhook events were lost. The processing backlog did not surface within the webhook workers until 5:06 AM ET on June26. Once it became available, the queued webhook events were processed, and webhook delivery latency returned to normal at approximately 6:03 AM ET. The pipeline auto-recovered without any manual intervention. Subsequent log and metric analysis identified an issue in the upstream message consumption layer as the primary contributor: following an infrastructure scale-up, the Kafka consumer group that feeds the webhook processing pipeline did not rebalance correctly, leaving a subset of partitions effectively unassigned for consumption. As a result, webhook events landing on those partitions were not picked up by the webhook workers in real time. The condition self-resolved when consumer group membership stabilized, at which point the queued events were processed quickly and deliveries returned to normal. Standard Kafka topic-lag monitoring did not surface the condition in a timely manner because aggregate topic lag remained within thresholds while the affected partitions accumulated work in the background. Socure engineering has identified a set of corrective and preventive actions — described below — to harden the webhook processing pipeline against partition rebalance failures and to improve observability of partition-level consumption health. As an immediate preventive action, an additional alert that detects this class of failure has already been deployed. **2. Timeline** | **Time \(ET\)** | **Event** | | --- | --- | | **Thu, Jun 25 — 2:06 PM** | Webhook delivery for a subset of events starts getting impacted. This was following an infrastructure scale-up performed, the Kafka consumer group that feeds the webhook processing pipeline did not rebalance correctly, leaving a subset of partitions effectively unassigned for consumption. Webhook events on those partitions begin to accumulate without being delivered. Standard Kafka topic-lag monitoring does not surface because aggregate topic lag remains within thresholds. | | **Thu, Jun 25 2:06 PM – Fri, Jun 26 – 5:06 AM** | Webhook delivery latency remains elevated for approximately half of webhook events. Events on unaffected partitions continue to be delivered normally. No webhook events are lost. | | **Fri, Jun 26 — 5:06 AM** | The processing backlog surfaces within the webhook workers. Consumer group membership stabilizes, the affected partitions begin to be consumed, and the webhook workers report a steep increase in lag as they begin processing the queued events. | | **Fri, Jun 26 — 5:06 AM –6:03 AM** | Queued webhook events are processed rapidly. Webhook delivery latency returns to normal at approximately 6:03 AM ET. No manual mitigation is required. | | **Fri, Jun 26 — 10:00 AM** | On-call engineering reviews telemetry from the overnight processing spike and correlates it with customer reports of delayed webhook deliveries. | | **Fri, Jun 26 — 11:57 AM** | Incident formally declared in Socure’s incident management system as a retrospective incident. Incident response team assembled and investigation bridge convened. Telemetry for the webhook processing pipeline reviewed in parallel. | | **Post Incident** | Detailed log and metric analysis of the webhook processing pipeline. Primary contributor identified as a Confluent Kafka | **3. Root Cause** **Primary Root Cause:** Following an infrastructure scale-up performed Thu, Jun 25 02:06 PM, the Kafka consumer group that feeds Socure’s webhook processing pipeline did not rebalance correctly. As a result, a subset of Kafka partitions was effectively unassigned for consumption, and webhook events landing on those partitions were not picked up by the webhook workers in real time. The condition was transient: when consumer group membership stabilized at approximately 5:06 AM ET on June 26, the affected partitions resumed consumption, the queued events were processed rapidly, and webhook delivery latency returned to normal at approximately 6:03 AM ET. No webhook events were lost. **Contributing Factors:** * **Consumer group rebalance failure:** The configured Kafka consumer rebalance protocol did not produce a complete partition assignment for the consumer group following the scale-up.Investigation revealed an edge-case consumer group rebalance failure during the infrastructure scale-up. We have since implemented application-level validation to actively monitor partition health.. * **Observability gap on partition-level consumption:** Existing Kafka topic-lag monitoring did not surface the condition in a timely manner. Topic-aggregate lag remained within thresholds during the impact window because lag on actively consumed partitions stayed at zero, masking the accumulating lag on the affected partitions until consumption resumed and the workers reported a step increase. **Non-contributing Factors:** * **Webhook worker fleet capacity:** The webhook worker fleet was operating within nominal capacity throughout the incident window. No worker-level resource saturation, restart, or scaling failure was associated with the incident. * **Customer-side delivery endpoints:** Customer-owned webhook endpoints were not associated with the incident. Once the backlog began processing, deliveries to customer endpoints completed at expected rates. * **Recent application code changes:** No Socure webhook application code release during the incident window was associated with the incident. **4. Resolution** **Service restoration:** The condition was transient and self-resolved when the affected Confluent Kafka consumer group’s partition assignment stabilized at approximately 5:06 AM ET on June 26, 2026. At that point, the previously unassigned partitions resumed consumption, and the webhook workers processed the queued events rapidly. Webhook delivery latency returned to normal at approximately 6:03 AM ET. No manual mitigation was required. Socure engineering verified recovery through telemetry and continued monitoring through and beyond the incident window. **Validation and verification:** A post-incident review was conducted to confirm the scope and duration of impact. Detailed log and metric analysis identified the Confluent Kafka consumer group rebalance failure as the primary contributor, occurring in the context of an infrastructure scale-up performed the prior day. As an immediate preventive action, Socure engineering deployed an additional alert that monitors divergence between message production rate and message polling \(consumption\) rate over a time period — designed specifically to detect scenarios where partitions are not being consumed even when topic-aggregate lag does not surface the delay. Additionally, we have now added a safety net for watching partition updates for ensuring the partition assignments. **5. Corrective and Preventive Actions** | **Action** | **Description** | **ETA / Status** | | --- | --- | --- | | **Partition-level consumption monitoring** | Deploy an additional alert that monitors divergence between message production rate and message polling \(consumption\) rate over a time period, complementing existing Kafka topic-lag monitoring to detect partition-level consumption gaps that aggregate lag alone does not surface. | **Completed** | | **Consumer group handling partitional updates** | We have implemented application-level event listeners to dynamically monitor Kafka partition state changes. In the event of a recurring Kafka broker or partition failure, the application will now automatically detect the disruption and execute a self-healing recovery sequence. | **June 26th, 2026** | | **Scale-up runbook hardening** | Update the infrastructure scale-up runbook for the webhook processing pipeline to include explicit verification of Kafka consumer group rebalance completion and partition assignment health before the scale-up is declared complete. | **mid-Q3, 2026** | **6. Lessons Learned** This incident exposed two specific gaps in the webhook processing pipeline. First, infrastructure scale-up operations can leave the Confluent Kafka consumer group in an incomplete rebalance state, with a subset of partitions effectively unassigned, without that condition surfacing to operators in real time. Second, Kafka topic-aggregate lag is not a sufficient indicator of partition-level consumption health when only a subset of partitions is affected, because lag on actively consumed partitions can mask accumulating lag on unassigned ones. Follow-up work is focused on detecting and preventing partition-level consumption gaps, hardening the scale-up runbook for the webhook processing pipeline, and strengthening observability so that rebalance anomalies are surfaced as they occur rather than after they have already produced customer-visible impact. **7. Next Steps & Ongoing Commitment** Socure takes accountability for this incident and the impact it had on the affected customers. We remain committed to improving the resilience and maintaining a reliable experience for our customers. The corrective actions underway focus on three areas: strengthening partition-level consumption monitoring for the webhook processing pipeline so that rebalance-related consumption gaps are detected as they occur, identifying and remediating the underlying cause of the Confluent Kafka consumer group rebalance failure, and hardening the infrastructure scale-up runbook to verify consumer group health and partition assignment before any scale-upis declared complete. We remain committed to improving the resilience and maintaining a reliable experience for our customers.

Read the full incident report →

Started Jun 23, 2026, 02:56 AM UTC· resolved Jun 23, 2026, 03:29 AM UTC

1. investigating Jun 23, 2026, 02:56 AM UTC

   We are currently experiencing errors affecting our RiskOS APIs. Our team is actively investigating the issue and will provide updates as more information becomes available.

2. monitoring Jun 23, 2026, 03:07 AM UTC

   Our team has mitigated the issue that was causing errors with the RiskOS APIs. We are seeing error rates return to normal levels and are continuing to monitor the system closely. We will provide additional updates if there are any further changes.

3. monitoring Jun 23, 2026, 03:24 AM UTC

   We are continuing to monitor our services and are not observing any errors at this time. We will provide additional updates if there are any further changes.

4. resolved Jun 23, 2026, 03:29 AM UTC

   The issue impacting RiskOS has been resolved, and service has returned to normal operation. Customers should no longer experience errors, degraded performance, or delays when accessing RiskOS services and workflows. Our teams have confirmed system stability and will continue to monitor performance closely. We appreciate your patience throughout this incident and apologize for the impact this outage may have had on your operations.

5. postmortem Jun 26, 2026, 09:07 PM UTC

   **Root Cause Analysis** **Incident:** RiskOS Evaluation API Elevated 502 Errors **Date Range:** June 22, 2026, 10:23 PM EDT – 10:59 PM EDT **Impact:** Customers calling the Socure RiskOS Evaluation APIs in production experienced an elevated rate of HTTP 502 \(Bad Gateway\) responses for approximately 36 minutes. **Status:** Resolved ## **1. Summary** On June 22, 2026, between approximately 10:23 PM and 10:59 PM EDT, Socure’s RiskOS Evaluation APIs in the production environment experienced an elevated rate of HTTP 502 errors. The disruption was caused by a misconfiguration in the aggregations engine — a component that enriches Evaluation API processing by computing velocity based signals across entity data. The misconfiguration was introduced during a configuration update as part of Local Graph GA readiness, and was not caught by existing validation controls. When the service attempted to use this configuration during evaluation processing an unexpected runtime exception occurred that triggered automated service restarts. As the affected services cycled, backend capacity became intermittently unavailable, causing requests during those periods to receive HTTP 502 \(Bad Gateway\) responses. Socure's automated monitoring detected the elevated error rate at **10:24 PM EDT** and the Engineering team began investigation immediately. The team mitigated the issue by disabling aggregations processing to stabilize traffic, remediated the invalid configuration, and validated recovery before re-enabling aggregations. Evaluation APIs returned to normal operation by **10:59 PM EDT**. Aggregations were re-enabled after validation at approximately **11:13 PM EDT**. Requests that received HTTP 502 responses during the incident window were not processed and no corresponding evaluation was created. Customers who experienced HTTP 502 responses between 10:23 PM and 10:59 PM EDT on June 22, 2026 should verify any intended evaluations and resubmit them if necessary. ## **2. Timeline** | **June 22, 2026 / Time** | **Event** | | --- | --- | | **10:22 PM EDT** | A configuration change in Entity Aggregations introduced an invalid value. | | **10:23 PM EDT** | Elevated HTTP 502 error rates begin on RiskOS Evaluation APIs in the production environment. | | **10:24 PM EDT** | Automated monitoring detected the elevated error rate and the on-call Engineering team was notified. Investigation began immediately. | | **10:25 PM to 10:40 PM EDT** | Engineering observed multiple servers failing intermittently. The team reviewed logs and traced the failures across affected servers to isolate the root cause. | | **~10:40 PM EDT** | Engineering identified that the issue was related to an aggregations configuration affecting Evaluation API processing. | | **~10:45 PM EDT** | Engineering began targeted mitigation, initially isolating specific aggregations-dependent components. Instability persisted, and the team proceeded to disable aggregations processing entirely to stabilize traffic. | | **10:55 PM EDT** | Aggregations processing was fully disabled. Evaluation API traffic stabilized. | | **10:59 PM EDT** | Evaluation APIs returned to normal operation. Elevated 502 errors were resolved. | | **10:59 PM to 11:13 PM EDT** | Engineering identified and remediated the specific invalid aggregation configuration. Recovery was validated against the corrected configuration. | | **11:13 PM EDT** | Aggregations re-enabled following successful validation of the corrected configuration. | ## **3. Root Cause** #### **Primary Root Cause:** An invalid configuration was applied to the aggregations engine. When the service attempted to use this configuration during Evaluation processing, it triggered an unhandled failure path inside the processing pipeline. Rather than gracefully rejecting the invalid configuration, the failure path caused service-level crashes and the affected servers entered a restart cycle. While the affected services were cycling, the RiskOS Evaluation APIs' available capacity to serve traffic dropped, and a portion of incoming requests reached the gateway without a healthy backend service available to respond. The gateway returned HTTP 502 \(Bad Gateway\) responses for those requests. #### **Contributing Factors:** * **Insufficient configuration validation:** The server did not gracefully reject the invalid Aggregation configuration at the load time. Instead of failing safely, the invalid spec propagated into a runtime failure path that crashed the affected pod, producing the customer-visible 502 responses. * **Shared processing impact:** Because the issue affected shared Evaluation API processing, a configuration-specific issue resulted in elevated API failures for customers during the incident window. #### **Non-contributing Factors:** * **Infrastructure health:** No underlying cluster, network, storage, or platform infrastructure issue contributed to the incident. Compute, network, and storage layers supporting RiskOS remained healthy throughout the incident window. Once the offending configuration was mitigated, the backend service stabilized with Evaluation processing restored back to normal. ## **4. Resolution** Socure restored service by disabling aggregations processing entirely, which stabilized Evaluation API traffic and returned normal operation by 10:59 PM EDT. The team then identified and removed the specific invalid aggregation configuration, validated the correction, and re-enabled aggregations at 11:13 PM EDT. Runtime safeguards have been implemented to handle invalid aggregation configurations gracefully without affecting service availability. Pre-activation validation checks currently in development will prevent invalid configurations from reaching production. ## **5. Corrective and Preventive Actions** | **Action** | **Description** | **ETA / Status** | | --- | --- | --- | | **Runtime safeguards for Aggregations Configurations** | Updated Evaluation API processing to handle incomplete or missing aggregation configuration values gracefully across all similar configuration processing paths, preventing service crashes. The specific condition that triggered this incident is now handled safely without impacting service stability. | **Completed** | | **Aggregations configuration validation** | Added stricter validation so invalid, incomplete, or unsupported configurations - are rejected before they can become active in production. | **In Progress** | | **Pre-activation configuration checks** | Added checks before aggregation configurations are used in Evaluation API processing to identify invalid or conflicting states earlier in the pipeline, before they can impact live traffic. | **In Progress** | | **Blast radius control** | Strengthen isolation so an issue in a specific workflow path does not affect broader shared Evaluation API processing. | **Q3, 2026** | ## **6. Lessons Learned** This incident highlighted a specific gap between Aggregations configuration validation and Evaluation API runtime handling. Follow-up work is focused on preventing invalid configurations from becoming active, improving runtime safeguards, and strengthening isolation for workflow-specific issues. ## **7. Next Steps & Ongoing Commitment** Socure recognizes that customers rely on RiskOS Evaluation APIs for reliable, real-time decisioning. We take ownership of the disruption caused by this incident and are implementing changes to reduce the likelihood and impact of similar issues. The corrective actions underway focus on three areas: strengthening aggregations configuration validation, improving Evaluation API runtime safeguards, and limiting the blast radius of any future configuration-related issues. We remain committed to improving the resilience and maintaining a reliable experience for our customers.

Read the full incident report →