Sardine Outage History

Started Feb 03, 2026, 04:29 PM UTC· resolved Feb 03, 2026, 04:29 PM UTC

1. monitoring Feb 03, 2026, 04:29 PM UTC

   Feb 3, 16:29 UTC Resolved - This incident has been resolved. Feb 3, 12:41 UTC Identified - We have identified a lag in transaction processing for the Sardine dashboard and are actively working on resolution.

Read the full incident report →

Started Jan 29, 2026, 08:20 AM UTC· resolved Jan 29, 2026, 08:20 AM UTC

1. monitoring Jan 29, 2026, 08:20 AM UTC

   Jan 29, 08:20 UTC Resolved - The issue has been resolved by the upstream provider, and latency has now returned to normal for Sanctions, PEP, and Adverse Media services. Jan 29, 05:49 UTC Update - We have observed a noticeable improvement in latency and are continuing to work closely with the provider to stabilize the service. The team is actively monitoring the situation and will share another update once latency has fully returned to normal levels. Jan 28, 23:28 UTC Update - Latency issues started 15:00 UTC Jan 28th Jan 28, 23:27 UTC Update - We are continuing to investigate this issue. Jan 28, 23:22 UTC Investigating - We've noticed increased latency and even timeouts. Our team is working with the provider on getting more information and fixing the problem.

Read the full incident report →

Started Jan 15, 2026, 07:09 PM UTC· resolved Jan 15, 2026, 07:09 PM UTC

1. monitoring Jan 15, 2026, 07:09 PM UTC

   Jan 15, 19:09 UTC Resolved - This incident has been resolved. Jan 15, 17:24 UTC Identified - The issue has been identified and a fix is being implemented. Jan 15, 16:27 UTC Investigating - Some PATCH requests being sent to our business verification endpoint might fail. We are already working with our providers for fixing the issue.

Read the full incident report →

Started Jan 13, 2026, 08:30 AM UTC· resolved Jan 13, 2026, 08:30 AM UTC

1. resolved Jan 13, 2026, 11:37 AM UTC

   Since January 13 2025 05:30 to January 13 2025 10:30 http://crypto.sardine.ai was innacessible. Backend crypto API was working fine.

Read the full incident report →

Started Jan 09, 2026, 10:48 PM UTC· resolved Jan 09, 2026, 10:48 PM UTC

1. monitoring Jan 09, 2026, 10:48 PM UTC

   Jan 9, 22:48 UTC Resolved - The start was 19:20 UTC and end was 19:45 am UTC. Jan 9, 19:52 UTC Investigating - The Dashboard is having intermittent timeouts. Our team is aware and working on the issue.

Read the full incident report →

Started Jan 02, 2026, 03:00 PM UTC· resolved Jan 02, 2026, 04:30 PM UTC

1. resolved Jan 02, 2026, 05:00 PM UTC

   Dear Customer, We experienced intermittent timeouts with our address verification service between 9:05 AM CST and 10:30 AM CST today. During this window, customers using the /v1/customers API endpoint with address verification enabled may have been impacted. Impact: - Requests may have experienced slightly increased latency - Address verification data may have been temporarily unavailable in API responses - Risk assessments continued to process, but may not have included address-based signals Status: Resolved / Monitoring The timeout issue was limited to the timeframe noted above. We are actively monitoring the service and continuing to work with our provider to ensure ongoing stability. Core risk assessment functionality remained operational throughout the incident. Please reach out at [email protected] if you have any questions or require further clarification.

Read the full incident report →

Started Dec 30, 2025, 09:14 PM UTC· resolved Dec 30, 2025, 09:36 PM UTC

1. resolved Dec 30, 2025, 09:55 PM UTC

   Between 21:14 UTC and 21:36 UTC the US Production Dashboard had intermittent timeouts. Our internal team worked on this issue internally and it's resolved.

Read the full incident report →

Started Dec 25, 2025, 06:30 PM UTC· resolved Dec 25, 2025, 06:30 PM UTC

1. resolved Dec 25, 2025, 07:38 PM UTC

   Between 18:32:38 AM UTC and 18:59:38 AM UTC our US Production dashboard couldn't be accessed (all other were working without any issues).

Read the full incident report →

Started Dec 17, 2025, 04:03 PM UTC· resolved Dec 17, 2025, 04:03 PM UTC

1. resolved Dec 17, 2025, 04:03 PM UTC

   Overview As part of a configuration change required by our bank enrichment data partner, there was a defect that resulted in a failing bank enrichment check. API response included reason code `BVERR` indicating error with bank verification. While Sardine tested this configuration change and performed staged rollout with monitoring, our testing and monitoring were insufficient and missed certain configuration combinations which resulted in this incident. Timeline - 2025-02-20: code defect was introduced, while problematic code path was not executed - 2025-12-11 14:13:40: rollout of feature to first group of client started - 2025-12-11 17:02:40: rollout of feature to second group of client started - 2025-12-11 20:14:17: rollout of feature to third group of client started - 2025-12-15 18:01:40:: rollout of feature to fourth group of client started - 2025-12-16 21:09:04: Bug identified and feature flag disabled to mitigate impact What went wrong - Insufficient test coverage: e2e tests didn't cover the specific configuration combination - Insufficient monitoring and rollout process: while we performed staged rollout and had standard monitors around HTTP response codes, we didn’t notice the issue since our data provider responded with 200 (OK) status code - Logging incident: coinciding with this rollout, we had separate incident where we were dropping some part of application logs. During the rollout we confirmed the absence of error logs, but that was due to the separate incident. Action items - Enhance automated monitors to include more granular error reason codes - Update the internal runbook for more robust monitoring process

Read the full incident report →

Started Dec 11, 2025, 09:25 PM UTC· resolved Dec 11, 2025, 09:25 PM UTC

1. monitoring Dec 11, 2025, 09:25 PM UTC

   Dec 11, 21:25 UTC Resolved - This incident has been resolved. Dec 11, 20:47 UTC Monitoring - A fix is being released gradually and we are monitoring the situation Dec 11, 18:30 UTC Identified - Issue is on Sardine side and team is already deploying a fix Dec 11, 17:45 UTC Investigating - Under certain circumstances our /v1/businesses endpoint could return an error an the following message: "This feature is not available for this account. Please contact [email] to enable this feature" We are working along the vendor to get it sorted out.

Read the full incident report →

Started Dec 08, 2025, 06:00 AM UTC· resolved Dec 08, 2025, 06:00 AM UTC

1. resolved Dec 08, 2025, 07:47 PM UTC

   Summary On December 8, 2025, Sardine's AML screening integration service experienced two outages affecting Sanctions / PEP / Adverse Media Screening. First Outage: Start Time: 2025-12-08 06:13:05 UTC End Time: 2025-12-08 08:56:05 UTC Second (minor) Outage (only a few transactions were affected during this long period): Start Time: 2025-12-08 13:58:26 UTC End Time: 2025-12-08 18:09:22 UTC What Happened? Sardine’s screening partner experienced an outage within their product resulting in some screening API responses being returned as NULL for the screening requests. During the incident, Sardine responded with reason code SSFD, indicating the screening failure. If you have an alert queue configured for this reason code, you may have seen an increased number of alerts because of the incident. We recommend to rerun all the events during this period for the correct screening results to an inquiry.

Read the full incident report →

Started Dec 04, 2025, 02:59 PM UTC· resolved Dec 04, 2025, 02:59 PM UTC

1. resolved Dec 04, 2025, 05:02 PM UTC

   Our third party provider for DocKYC experienced a service interruption on their platform that affected their Onboarding and Authentication services. The service has been fully restored and they're continuing to actively monitor the situation to ensure the stability of their systems. Start time: Dec 04 02:59:10 PM UTC End time: Dec 04 03:13:54 PM UTC Total duration: 14 minutes

Read the full incident report →

Started Nov 29, 2025, 07:20 PM UTC· resolved Nov 29, 2025, 07:20 PM UTC

1. monitoring Nov 29, 2025, 07:20 PM UTC

   Nov 29, 19:20 UTC Resolved - The issue has been fully resolved, and all systems are operating normally. Nov 29, 18:56 UTC Monitoring - Service has been restored as of 18:45 UTC. We are closely monitoring system performance to ensure stability. Nov 29, 18:43 UTC Investigating - We are currently investigating an outage affecting system availability. Further updates will be provided as they become available.

Read the full incident report →

Started Nov 28, 2025, 06:30 AM UTC· resolved Nov 28, 2025, 07:20 PM UTC

1. resolved Nov 28, 2025, 07:44 AM UTC

   Dear Customer, We experienced a brief period of intermittent access on 28th Nov 2025, between 06:43 UTC and 06:48 UTC. Our team quickly identified the root cause and has implemented system reinforcements to prevent recurrence. All services have been fully restored and are operating as expected. Should you notice any further anomalies, please report them to us immediately.

2. postmortem Dec 04, 2025, 11:50 AM UTC

   # Post-Incident Report: Service Degradation \(November 28 & 29, 2025\) _\(All timestamps are in UTC\)_ ## Summary * **First Incident \(Nov 28\):** Elevated error rates and intermittent access were observed for approximately **5 minutes** \(06:43–06:48 UTC\). * **Second Incident \(Nov 29\):** **70-minute window** of elevated errors \(18:10–19:20 UTC\), including a severe service degradation period of roughly **22 minutes**. ## Root Cause This was caused by an unexpected and significant spike in traffic volume. The surge in requests temporarily exceeded our forecasted capacity and our auto scaling capability, causing congestion in our application layer.Impact ## Symptoms During these windows, customers using the Dashboard and Core Risk APIs experienced increased latency and 502/503/504 errors.Resolution and Next Steps ## Short Term Solution Our engineering teams intervened to stabilize the platform during the events. ## Long Term Solution To ensure our systems remain resilient against future traffic spikes of this magnitude, we are currently provisioning additional infrastructure and permanently increasing our system capacity.

Read the full incident report →

Started Nov 18, 2025, 04:44 PM UTC· resolved Nov 18, 2025, 04:44 PM UTC

1. monitoring Nov 18, 2025, 04:44 PM UTC

   Nov 18, 16:44 UTC Resolved - Cloudfare fixed the issue on their end and we see no latency/errors from our providers now. Nov 18, 12:20 UTC Investigating - Due to Cloudfare global outage (https://www.cloudflarestatus.com/incidents/8gmgl950y3h7) expect latency and lack of information on the following external enrichments: - Remote Desktop Detection (only Anydesk) - Address enrichment - IP location - Crypto Enrichment - ID Verification - Sanctions, PEP, Adverse Media

Read the full incident report →

Started Oct 29, 2025, 07:28 PM UTC· resolved Oct 29, 2025, 07:49 PM UTC

1. investigating Oct 29, 2025, 07:28 PM UTC

   Team is investigating we'll update as soon as possible. Except errors from our API.

2. resolved Oct 29, 2025, 07:49 PM UTC

   We have had elevated 5xx errors for customers API from 19:04 UTC to 19:43 UTC incident is resolved now and we'll come back with post mortem as soon as possible.

3. postmortem Oct 31, 2025, 06:47 PM UTC

   # Rule engine outage on deploy ## Overview _On Oct 29, Sardine experienced an outage on our rules-engine service that impacted our_ `/v1/customers`endpoint causing some requests to fail during this period, with a big spike when the error happened and a fast decrease on the error rate. ## What happened Sardine deploys the backend services every Wednesday, during one of our regular deployments we noticed an error while the rules-engine service was being deployed in canary mode \(25% of traffic gets routed to the new version instances\). After noticing the error our team immediately started a rollback and checked for root causes, finding it a few minutes later and only continuing the deployment on the next day, this time, a successful one. The error was a database migration that caused the application old versions to lose the reference to schema they used, causing the errors and triggering 5xx responses in the `customers` api. ## Impact `/v1/customers` API endpoint responding with 500 http response for around 30 minutes, with the first minutes concentrating 90% of the errors ## Timeline \(UTC\) * 18:44 deployment ticket gets approved * 18:55 Release engineering starts deployment * 19:05 Release engineer notices something is wrong, rules-engine service has an elevated number of errors after deployment * 19:07 Release engineer starts rollback action * 19:10 incident is initiated internally and externally * 19:44 All services have finished rolling back to the previous version * 19:54 Incident is considered as solved. ## Action items * Fix root cause issue where migrations running in a canary deployment made old versions receive an error in prepared statement cache * Enhance internal toolings to deployment to have faster deployment and rollbacks

Read the full incident report →

Started Oct 20, 2025, 10:00 AM UTC· resolved Oct 20, 2025, 10:00 AM UTC

1. resolved Oct 20, 2025, 11:04 AM UTC

   Not a direct Sardine issue nor we were affected directly but AWS had an issue in N. Virginia zone: https://health.aws.amazon.com/health/status

Read the full incident report →

Started Sep 26, 2025, 01:27 PM UTC· resolved Sep 26, 2025, 06:35 PM UTC

1. investigating Sep 26, 2025, 01:27 PM UTC

   We're encountering an issue internally with loading some features like Backtesting and embedded charts in the Dashboard. Our engineering team is currently working on fixing this issue.

2. resolved Sep 26, 2025, 06:35 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 19, 2025, 06:45 PM UTC· resolved Sep 19, 2025, 08:31 PM UTC

1. identified Sep 19, 2025, 06:45 PM UTC

   Some Sandbox rules might not be appearing under the Customer checkpoint. Timestamp of when the issue started on: 2025-09-19 01:06 AM UTC Our team identified the issue and is working towards a fix.

2. resolved Sep 19, 2025, 08:31 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 19, 2025, 02:00 AM UTC· resolved Sep 20, 2025, 03:16 PM UTC

1. resolved Sep 19, 2025, 04:47 PM UTC

   We encountered an issue executing workflows via the Customers API, between Sept 18th 11:10pm UTC to Sept 19th 3:16 PM UTC. The team performed a fix for the issue, and customer's workflows should be working as expected again.

Read the full incident report →

Started Sep 17, 2025, 10:55 AM UTC· resolved Sep 17, 2025, 11:45 AM UTC

1. resolved Sep 17, 2025, 04:44 PM UTC

   Our third party provider for Bank enrichment encountered an outage on their end, remaining unavailable for 40 minutes with timeouts. During the outage, the system returned HTTP  500 responses for about 5 minutes. After 40 minutes of remediation, their system recovered. The issue started 10:55 AM UTC and ended on 11:45 AM UTC.

Read the full incident report →

Started Sep 09, 2025, 06:17 PM UTC· resolved Sep 09, 2025, 08:22 PM UTC

1. identified Sep 09, 2025, 06:17 PM UTC

   In some cases, the API is having an instability and returning a 503 error. Some endpoints that may have this behavior are: /v1/customers, /v1/feedbacks, v1/issuing/risk. Our team is working on a fix for this issue.

2. resolved Sep 09, 2025, 08:22 PM UTC

   This incident has been resolved.

3. postmortem Sep 10, 2025, 12:35 PM UTC

   ### **Introduction** * **Purpose:** This report provides an overview of the recent service disruption impacting a few of our API endpoints * **Apology:** We sincerely apologize for any inconvenience this disruption may have caused. We remain dedicated to maintaining high service availability and reliability. ### **Incident Overview** * **Duration:** 1 Minute around 17:36 UTC and 1 minute around 17:56 UTC. * **Region Affected:** US * **Services Affected:** Customers, Feedbacks and Issuing Risk API endpoints ### **Root Cause Analysis** * **Primary Issue:** An overlooked edge case in the handler of a new API being rolled out internally * **Detailed Explanation:** The issue started occurring intermittently after we enabled a feature flag that replaces one API used internally by our dashboard. The handler of the new API did not account for one edge case that happened infrequently but which led pods running our application to crash. Even though this feature flag was exposed to very few clients, the pod crashing was enough to disrupt unrelated services. ### **Impact** * Higher number of 503 responses from endpoints like `/v1/customers`, `/v1/feedbacks`, `/v1/issuing/risk` ### **Detection and Recovery Time** * **Why this wasn’t noticed earlier:** The edge case in question wasn’t covered in tests. Once the issue was identified in prod, we immediately diagnosed and fixed it. ### **Corrective Actions and Improvements** * **Immediate Response:** The feature was disabled in production. * **Preventive Measures:** We’ve fixed the handler of the new API and expanded our test coverage to ensure this cannot happen again. ### **Conclusion** * **Commitment:** Sardine remains firmly committed to delivering reliable and resilient services to our partners. We deeply regret the inconvenience caused by this incident and appreciate your patience and understanding. * **Appreciation:** Thank you for your continued trust and partnership. We value your support as we strengthen our systems and processes to ensure greater reliability and stability.

Read the full incident report →

Started Aug 13, 2025, 11:00 PM UTC· resolved Aug 13, 2025, 11:00 PM UTC

1. resolved Aug 14, 2025, 12:04 PM UTC

   While deploying a new component to display Synthetic ID information we changed the way the backend and frontend communicates which brought issues. - This did not cause any data loss/risk analysis errors, it only prevented clients from opening specific customer sessions. - All sessions can be visualized normally after the fix was deployed. (around 11 PM UTC) - The problem started with the new Backend version deployment, around 8 PM UTC, and was fixed around 11 PM UTC.

Read the full incident report →

Started Aug 01, 2025, 04:03 PM UTC· resolved Aug 04, 2025, 12:17 PM UTC

1. investigating Aug 01, 2025, 04:03 PM UTC

   We are currently encountering an issue with a search degradation on Customer Intelligence, some columns in the Customer Intelligence view are also missing at the moment. Feedback status can also be outdated. Manual Verification is also not available. Our engineering team is working on a fix.

2. identified Aug 01, 2025, 07:19 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Aug 01, 2025, 09:25 PM UTC

   These environments are back to normal: - Prod US - Prod EU - Sandbox EU Team is still working on Sanbox US

4. resolved Aug 04, 2025, 12:17 PM UTC

   This incident has been fully resolved. Sandbox US search was re enabled during the weekend.

Read the full incident report →

Started Jul 24, 2025, 08:41 PM UTC· resolved Jul 24, 2025, 08:53 PM UTC

1. identified Jul 24, 2025, 08:41 PM UTC

   The issue has been identified and a fix is being implemented.

2. identified Jul 24, 2025, 08:43 PM UTC

   Team already found root cause and it's working on a fix.

3. monitoring Jul 24, 2025, 08:50 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Jul 24, 2025, 08:53 PM UTC

   This incident has been resolved.

5. postmortem Jul 25, 2025, 01:41 AM UTC

   # Incorrect Very High Risk Evaluations **Date:** 2025-07-24 **Duration:** 48 minutes **Severity:** High **Impact:** Multiple B2B clients affected ## Executive Summary A code deployment introduced a race condition in our risk evaluation query logic, causing legitimate sessions to be incorrectly flagged as "very\_high" risk. The incident affected multiple clients for 48 minutes before detection and resolution. ## Incident Details ### What Happened Sessions across multiple Sardine clients received elevated risk scores \("very\_high"\) that did not align with their actual risk profile as determined by our checkpoint-specific rules and ML models. Root cause: a race condition where risk queries executed before the risk engine completed session evaluation, triggering fail-safe logic inappropriately. ### Timeline \(UTC\) | Time | Event | Actor | | --- | --- | --- | | 20:02 | Canary \(phased\) deployment initiated with affecting change | Engineering | | 20:17 | Automated promotion to full deployment | CI/CD Pipeline | | 20:19 | **First client escalation** - elevated risk levels reported | Client | | 20:29 | Issue escalated to engineering | Support Team | | 20:38 | Root cause confirmed, rollback initiated | Engineering | | 20:50 | **Service fully restored** | Engineering | **Key Metrics:** * Time to Detection: 17 minutes \(customer-reported\) * Time to Resolution: 31 minutes from escalation * Total Impact Duration: 48 minutes ## Five Whys Analysis 1. Why were sessions incorrectly marked as very\_high risk? * Risk query executed before risk engine evaluation completed, triggering fail-safe "default to block" logic 1. Why did the risk query execute before risk engine completion? * Race condition introduced in deployment caused timing dependency failure between risk evaluation and query execution 1. Why wasn't this race condition detected in testing? * QA environment lacked sufficient session volume to reproduce the timing conditions that trigger the race condition 1. Why didn't our monitoring detect this anomaly? * Missing alerting thresholds for sudden increases in very\_high risk session percentages per client 1. Why didn't phased rollout catch this issue? * Only subset of clients affected due to configuration differences; clients with custom risk logic bypassed the affected code path ## Impact Analysis ### What Went Wrong 1. **Detection Failure**: No proactive monitoring detected the anomaly - relied on customer escalation 2. **Testing Gap**: QA missed race condition scenarios due to insufficient test data volume 3. **Monitoring Gap**: No alerting on risk level distribution changes 4. **Phased Launch Limitation**: Configuration variance masked issue during gradual rollout ### What Went Well 1. **Rapid Response**: Support team escalated within 10 minutes of customer report 2. **Quick Resolution**: Engineering confirmed and initiated rollback within 19 minutes 3. **Effective Rollback**: Clean restoration with no data corruption ## Action Items ### Immediate \(Week 1\) * \[ \] **Owner: Engineering Lead** - Implement alerting for anomalous \(zscore\) increase in very\_high risk sessions per client \(Target: 3 days\) * \[ \] **Owner: QA Lead** - Add race condition test scenarios with production-like session volumes \(Target: 5 days\) * \[ \] Per client incident report and queries for assessing impacted sessions ### Short-term \(Month 1\) * \[ \] **Owner: Platform Team** - Review and refactor fail-safe logic to handle timing dependencies \(Target: 2 weeks\) * \[ \] **Owner: Engineering** - Enhance unit and E2E test coverage for risk evaluation edge cases \(Target: 3 weeks\) * \[ \] **Owner: QA Team** - Implement exploratory testing protocols for risk evaluation scenarios \(Target: 4 weeks\) ### Medium-Term In progress - ownership Engineering ## Prevention Measures 1. **Enhanced Monitoring**: Real-time risk distribution tracking with client-level granularity 2. **Improved Testing**: Production-volume simulation in staging environment 3. **Robust Fail-safes**: Timeout handling and graceful degradation for race conditions 4. **Phased Deployment**: Configuration-aware rollout strategy to ensure comprehensive coverage ## Lessons Learned * Customer-reported incidents indicate monitoring blind spots requiring immediate attention * Race conditions in distributed systems require specific testing strategies beyond happy/sad path scenarios * Fail-safe mechanisms must account for timing dependencies in microservices architectures **Document Owner:** Engineering Team **Review Date:** 2025-08-24 **Distribution:** Executive Team, Engineering, Support, QA

Read the full incident report →