Spreedly Outage History

Started Sep 02, 2026, 03:11 PM UTC· resolved Sep 02, 2026, 09:32 PM UTC

1. investigating Sep 02, 2026, 03:11 PM UTC

   Spreedly has detected an issue potentially resulting in degraded performance on transactional APIs. While we are providing an early notification in an effort to alert you as quickly as possible, we are still investigating the actual scope and impact and will provide an update as soon as more details are available. Thank you for your patience.

2. monitoring Sep 02, 2026, 03:57 PM UTC

   We’re working to resolve a performance lag affecting some payment processing (specifically purchase, capture, and payment method creation) related to recent database maintenance. A fix is underway, and our team is actively monitoring the system. Please check back here for real-time updates.

3. resolved Sep 02, 2026, 09:32 PM UTC

   This incident has been resolved.

4. postmortem Sep 09, 2026, 09:50 PM UTC

   ### _September 9, 2026_ — _Elevated Error Rates Affecting Transaction Processing_ ### ‌High Level Summary On September 2, 2026, between 14:35 and 15:19 UTC, Spreedly experienced a period of elevated error rates affecting a subset of transaction requests across endpoints. The issue lasted approximately 44 minutes and was caused by an infrastructure-level issue with our database platform. Our team identified and resolved the issue the same day. ### What Happened During a routine schema update, a defect in our underlying database platform caused one of our database nodes to become unavailable. This temporarily reduced the redundancy for a portion of our transaction data, which led some requests to either time out or receive a system error response \(HTTP 500\) instead of processing normally. Our team worked directly with our database vendor to identify the root cause and confirm full recovery of the affected infrastructure. ### Next Steps We've identified the underlying cause as a known issue in our database vendor's software. A vendor-supplied fix is scheduled to be applied during normal scheduled maintenance. In parallel, we are adding additional monitoring to detect this class of issue earlier, and adjusting our internal procedures around database maintenance work to further reduce the risk of a similar incident going forward. -The Spreedly Team

Read the full incident report →

Started Aug 14, 2026, 04:14 PM UTC· resolved Aug 14, 2026, 04:34 PM UTC

1. investigating Aug 14, 2026, 04:14 PM UTC

   We are currently investigating an increase in 5xx errors on the Core Transactional API. Some transactions may be failing or returning unexpected error responses during this time. We will provide an update as soon as we have more information.

2. resolved Aug 14, 2026, 04:34 PM UTC

   A fix has been implemented and error rates have returned to normal. Some transactions may have failed or returned errors during the affected window. We're continuing to monitor and will follow up with a more detailed post-incident summary.

3. postmortem Sep 03, 2026, 08:55 PM UTC

   ### Service Disruption on Transaction Endpoints ### ‌High Level Summary On August 14, 2026, Spreedly experienced a brief service disruption affecting some transaction-related API endpoints for approximately 20 minutes \(4:06 PM – 4:26 PM ET\). During this window, some requests to endpoints involving transaction lookups, captures, voids, credits, and payment method operations may have received error responses. The issue was identified and resolved quickly by our engineering team. ### What Happened A routine database schema update caused a subset of existing database connections to reference outdated schema information, resulting in intermittent errors on affected requests. Once identified, our team refreshed the affected database connections, which fully resolved the issue. ### Next Steps We are reviewing our schema migration process to add additional safeguards that will prevent this class of issue from recurring, and are improving automated testing coverage for this scenario. We have added auto-retry and recovery mechanisms so that if a connection does reference an outdated schema it will not affect transactions going forward.

Read the full incident report →

Started Jul 27, 2026, 01:50 PM UTC· resolved Jul 27, 2026, 09:41 PM UTC

1. investigating Jul 27, 2026, 01:50 PM UTC

   We are currently investigating this issue.

2. identified Jul 27, 2026, 02:00 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Jul 27, 2026, 03:00 PM UTC

   Spreedly detected an issue that may have impacted core transaction performance. While the issue now appears to be resolved, our team is actively monitoring systems to ensure continued stability. We will provide a final update once our investigation into the full scope and impact is complete. Please continue to follow our StatusPage for the latest update.

4. resolved Jul 27, 2026, 09:41 PM UTC

   After closely monitoring and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We are completing our investigation concerning the causes of the incident and any residual impact. We apologize for any inconvenience or disruption.

5. postmortem Sep 03, 2026, 08:57 PM UTC

   ### High Level Summary On July 27, 2026, Spreedly experienced intermittent errors on a subset of transaction requests, including purchases, authorizations, and webhooks. ### What Happened A database migration updated the schema of an internal service that manages stored payment methods. This caused errors on existing connections referencing a stale cache. We rolled back the change and cycled those connections, fully resolving the issue. ### Next Steps We are reviewing our schema migration process to add additional safeguards that will prevent this class of issue from recurring, and are improving automated testing coverage for this scenario. We have added auto-retry and recovery mechanisms so that if a connection does reference an outdated schema it will not affect transactions going forward.

Read the full incident report →

Started Jul 11, 2026, 02:33 PM UTC· resolved Jul 11, 2026, 03:38 PM UTC

1. investigating Jul 11, 2026, 02:33 PM UTC

   We are investigating elevated error rates affecting certain API requests. Some list and retrieval endpoints may return intermittent 500 responses while we work to identify the root cause.

2. resolved Jul 11, 2026, 03:38 PM UTC

   We identified and resolved the issue causing intermittent 500 responses on certain list and retrieval endpoints. A configuration update has been deployed to prevent requests from being prematurely terminated, and we are continuing to monitor the service.

Read the full incident report →

Started Jul 08, 2026, 03:22 PM UTC· resolved Jul 08, 2026, 04:54 PM UTC

1. investigating Jul 08, 2026, 03:22 PM UTC

   Spreedly has detected an issue potentially resulting in degraded performance on Cybersource Credit transactions. While we are providing this early notification in an effort to alert you as quickly as possible, we are still investigating the actual scope and impact. We will provide an update as soon as more details are available.

2. identified Jul 08, 2026, 04:08 PM UTC

   We have identified transient internal DNS resolution error that occurred between 15:00 and 15:05 UTC causing select 500 errors across Spreedly services. We continue to investigate the full impact and will provide additional information as it becomes available.

3. resolved Jul 08, 2026, 04:54 PM UTC

   After closely monitoring and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We are completing our investigation concerning the causes of the incident and any residual impact. We apologize for any inconvenience or disruption.

Read the full incident report →

Started Jun 17, 2026, 02:55 PM UTC· resolved Jun 17, 2026, 09:42 PM UTC

1. investigating Jun 17, 2026, 02:55 PM UTC

   Spreedly has detected an issue potentially resulting in degrated performance with asynchronous transactions. While we are providing an early notification in an effort to alert you as quickly as possible, we are still investigating the actual scope and impact and will provide an update as soon as more details are available. Thank you for your patience.

2. monitoring Jun 17, 2026, 03:00 PM UTC

   We’ve implemented a fix and stabilized the asynchronous transactions. Our team is monitoring the situation closely to ensure there are no additional impacts. Please continue to follow our StatusPage for the latest update.

3. monitoring Jun 17, 2026, 04:03 PM UTC

   We are monitoring closely for any further issues or additional impacts. Please continue to follow our StatusPage for the latest updates.

4. resolved Jun 17, 2026, 09:42 PM UTC

   After a period of monitoring, we have confirmed that the issue has been resolved and systems are operating normally. If you continue to experience any issues, please reach out to Support.

5. postmortem Jun 24, 2026, 08:10 PM UTC

   June 24, 2026 — Asynchronous Transaction Processing Disruption **High Level Summary** On June 17, 2026 from approximately 10:02 AM to 10:41 AM UTC, Spreedly experienced a disruption affecting asynchronous gateway transactions and 3DS authentication flows. During the 39-minute window, approximately 1,634 transactions were left in a pending state without a confirmed outcome, representing less than 1% of total transaction volume during that period. **What Happened** A code deployment introduced a compatibility issue between newly deployed and previously running application instances. During a rollback process, some requests were routed across mixed code versions, causing failures when processing asynchronous transaction status updates. Affected transaction types include asynchronous gateway transactions \(gateway specific 3DS, and other asyncronous verifications, authorizations, and purchases\) and 3DS Global authentication. Transactions that entered the affected flow during this window may reflect an indeterminate outcome, the transaction state at your gateway may differ from what Spreedly reports. **Next Steps** We recommend reviewing transactions from the 10:02–10:41 AM UTC window on June 17 that remain in a pending state: Verifications may be safely retried. Authorizations should be voided at the gateway before retrying. Purchases carry a risk of duplication, we recommend refunding before retrying. Please reach out to Spreedly support if you need assistance identifying affected transactions or determining the appropriate remediation path. We are actively working to prevent this class of issue through improved deployment safeguards and automated rollback detection. -The Spreedly Team

Read the full incident report →

Started May 08, 2026, 06:00 PM UTC· resolved May 08, 2026, 06:05 PM UTC

1. monitoring May 08, 2026, 06:00 PM UTC

   A fix has been implemented and we are monitoring results.

2. resolved May 08, 2026, 06:05 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 19, 2026, 03:13 PM UTC· resolved Mar 19, 2026, 04:09 PM UTC

1. identified Mar 19, 2026, 03:13 PM UTC

   We are currently investigating performance of the Dodgeball service.

2. identified Mar 19, 2026, 03:30 PM UTC

   We are working to resolve an issue with delayed processing of requests to the Dodgeball services.

3. monitoring Mar 19, 2026, 03:55 PM UTC

   We’ve implemented a fix and stabilized the Dodgeball component that was experiencing a partial outage. We are continuing to monitor

4. monitoring Mar 19, 2026, 04:07 PM UTC

   After closely monitoring and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We apologize for any inconvenience or disruption.

5. resolved Mar 19, 2026, 04:09 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 16, 2026, 05:45 PM UTC· resolved Mar 16, 2026, 06:07 PM UTC

1. investigating Mar 16, 2026, 05:45 PM UTC

   We are currently investigating elevated errors on API endpoints

2. monitoring Mar 16, 2026, 05:51 PM UTC

   A fix has been implemented and we are monitoring to ensure services are operating correctly

3. resolved Mar 16, 2026, 06:07 PM UTC

   After closely monitoring and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We are completing our investigation concerning the causes of the incident and any residual impact. We apologize for any inconvenience or disruption.

4. postmortem Mar 19, 2026, 07:07 PM UTC

   ### Mar 18th 2026 — _Decryption Errors on API endpoints_ ### Summary Between 5:40 PM and 5:43 PM UTC on March 16, 2026, we experienced a brief issue during a key rotation that caused a small subset of API requests to fail due to decryption errors. These errors occurred within our system before any external processing took place. The issue was quickly identified, and traffic was immediately redirected to a stable environment, restoring normal service. ### What Happened At 5:36 PM UTC, we routed traffic to a newly deployed cluster of our encryption service running with the latest security keys. Shortly after, we observed decryption errors affecting a small subset of records, which resulted in some requests returning 500 error responses. We promptly redirected traffic back to the existing cluster, restoring normal operation. Further investigation determined that these records were still associated with a key in the process of rotation, and we updated them as a follow-up action to prevent the issue from recurring. ### Next Steps We have enhanced our process to ensure that a key’s usage is fully verified before it is removed from the encryption service. -The Spreedly Team

Read the full incident report →

Started Feb 26, 2026, 04:45 PM UTC· resolved Feb 26, 2026, 06:44 PM UTC

1. investigating Feb 26, 2026, 04:45 PM UTC

   We are currently investigating an issue affecting customers using Dodgeball. The production database backing Dodgeball has reached critically low available storage, which is expected to result in elevated error rates. Customers may begin experiencing errors within the next 10 minutes as the system’s ability to process requests degrades. Our engineering team is actively investigating and working to restore normal capacity as quickly as possible. We will provide further updates as more information becomes available.

2. monitoring Feb 26, 2026, 04:59 PM UTC

   The engineering team has successfully resolved the storage issue affecting the Dodgeball production PostgreSQL instance. Capacity has been restored and the system is now operating normally. At this time, we are not observing any elevated error rates for customers using Dodgeball. We will continue to monitor the system closely to ensure stability. Further updates will be provided if necessary.

3. resolved Feb 26, 2026, 06:44 PM UTC

   The storage issue affecting the Dodgeball production instance has been fully resolved. Capacity has been restored and systems are operating normally. We are no longer observing elevated error rates for customers using Dodgeball. Thank you for your patience while we worked to resolve this issue.

Read the full incident report →

Started Feb 05, 2026, 03:56 PM UTC· resolved Feb 05, 2026, 05:01 PM UTC

1. investigating Feb 05, 2026, 03:56 PM UTC

   We are currently investigating an issue where credit transactions referencing Capture transactions are failing in production. Some customers may see errors when attempting to issue credits against Capture transactions. Capture transactions themselves are processing successfully. Our engineering team is actively investigating this behavior to determine the cause and confirm whether this is a regression or an unintended change. We will provide updates as soon as more information is available.

2. monitoring Feb 05, 2026, 04:25 PM UTC

   We’ve deployed a revert to address the issue where credit transactions referencing Capture transactions were failing. We’re actively monitoring to ensure credit requests referencing Captures are processing as expected. At this time, we’re not seeing further errors related to this behavior.

3. resolved Feb 05, 2026, 05:01 PM UTC

   After monitoring the system post-deploy, we’re no longer seeing errors related to credits referencing Capture transactions, and processing has returned to normal. Thank you for your patience.

4. postmortem Feb 10, 2026, 05:22 PM UTC

   ### February 10, 2026 — Credits Referencing Capture Transactions Failing ### ‌High Level Summary Between approximately 1:56 PM UTC and 4:15 PM UTC on February 5, 2026, credit transactions where the reference transaction was a capture failed with the error "The specified `reference_token` corresponds to a transaction which cannot be used as a reference." ### What Happened At approximately 1:56 PM UTC on February 5, 2026, Spreedly rolled out a change to support credits on the CyberSource Rest gateway. This included a change in the way we validated reference transaction types used in credits. Specifically, there was an unintended side-effect which caused captures to no longer be treated as a valid reference type for credits on gateways other than CyberSource Rest. Once this was discovered, the code was reverted at 4:01PM UTC. By 4:15 PM UTC, credit requests against captures were fully functional. ### Next Steps * Any credit attempts that failed with this error are safe to retry * We are making improvements to our code review processes and regression test suite to ensure we catch similar errors before they make it into production * We have identified a gap in our observability surrounding validation errors that we will be prioritizing ‌ -The Spreedly Team

Read the full incident report →

Started Feb 04, 2026, 06:15 PM UTC· resolved Feb 04, 2026, 07:56 PM UTC

1. investigating Feb 04, 2026, 06:15 PM UTC

   Spreedly has detected an issue that is affecting multiple customers using Payflow Pro where some transactions fail with the error: “Invalid Request. Malicious string found in XML document”. While we are providing an early notification in an effort to alert you as quickly as possible, we are still investigating the actual scope and impact and will provide an update as soon as more details are available. Thank you for your patience.

2. monitoring Feb 04, 2026, 07:52 PM UTC

   We are no longer seeing errors at this time, and the issue appears to have subsided. We are working with Payflow Pro to identify the root cause. Our team continues to monitor the situation closely to ensure stability. Please continue to follow our StatusPage for the latest updates.

3. resolved Feb 04, 2026, 07:56 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 27, 2026, 07:59 PM UTC· resolved Jan 27, 2026, 08:12 PM UTC

1. investigating Jan 27, 2026, 07:59 PM UTC

   Spreedly has detected an issue related to a dodgeball component experiencing a partial outage. While we are providing an early notification in an effort to alert you as quickly as possible, we are still investigating the actual scope and impact and will provide an update as soon as more details are available. Thank you for your patience.

2. resolved Jan 27, 2026, 08:12 PM UTC

   We’ve implemented a fix and stabilized the Dodgeball component that was experiencing a partial outage. After closely monitoring and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We apologize for any inconvenience or disruption.

Read the full incident report →

Started Jan 27, 2026, 07:08 PM UTC· resolved Jan 27, 2026, 07:29 PM UTC

1. investigating Jan 27, 2026, 07:08 PM UTC

   Spreedly has detected an issue potentially resulting in a high number of pending 3DS2 Global transactions. While we are providing an early notification in an effort to alert you as quickly as possible, we are still investigating the actual scope and impact and will provide an update as soon as more details are available. Thank you for your patience.

2. monitoring Jan 27, 2026, 07:25 PM UTC

   We’ve implemented a fix and stabilized the 3DS2 Global transactions. Our team is monitoring the situation closely to ensure there are no additional impacts. Please continue to follow our StatusPage for the latest update.

3. resolved Jan 27, 2026, 07:29 PM UTC

   After closely monitoring and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We are completing our investigation concerning the causes of the incident and any residual impact. We apologize for any inconvenience or disruption.

4. postmortem Feb 03, 2026, 03:38 PM UTC

   ### February 3rd, 2026 — 3DS2 Global Transactions Pending ‌ Between 5:40 pm UTC and 7:24 PM UTC on January 27, 2026 the latest version of the Spreedly iFrame \(`iframe-v1`\) was configured to use version 1.180 which contained a bug related to the handling of device fingerprinting for `Sca::Authentications` in an end user's browser. As a result there was an elevated number of `Sca::Authentications` stuck in a pending flow with a `required_action` of `device_fingerprint`. These transactions were held in a pending state and never completed. Our team identified the issue and updated the version of iFrame \(`iframe-v1`\) to utilize `1.179`. ### What Happened At 5:40 pm UTC on January 27th, 2026 a new version of iFrame was released \(1.180\) and updated the version to be used in the `iframe-v1.min.js` asset to point to `1.180`. This change was meant to improve the error handling for the loading of 3rd party critical SDKs however it caused the SDK to enter a permanently error-ed state when attempting to perform device fingerprinting due to a race condition with loading the SDKs. At 7:20 pm UTC on January 27th, 2026 the change was rolled back and by 7:24 pm UTC, 2026 the `iframe-v1.min.js` asset globally was pointing to version `1.179`. ### Next Steps We have identified a gap in our observability and alerting within the iFrame service related to the managing of 3DS lifecycle flows that we will be prioritizing. -The Spreedly Team

Read the full incident report →

Started Jan 09, 2026, 06:46 PM UTC· resolved Jan 09, 2026, 07:10 PM UTC

1. investigating Jan 09, 2026, 06:46 PM UTC

   We are currently investigating this issue.

2. investigating Jan 09, 2026, 06:52 PM UTC

   We are continuing to investigate this issue.

3. monitoring Jan 09, 2026, 07:07 PM UTC

   A fix has been deployed and we are seeing systems recover. We will continue to monitor to ensure stability and will provide further updates as needed.

4. resolved Jan 09, 2026, 07:10 PM UTC

   After a period of monitoring, we have confirmed that the issue has been resolved and systems are operating normally. If you continue to experience any issues, please reach out to Support.

Read the full incident report →

Started Dec 28, 2025, 12:00 AM UTC· resolved Dec 28, 2025, 06:00 AM UTC

1. resolved Dec 29, 2025, 04:50 PM UTC

   Between 7:00 PM EST on Saturday, December 27th, and 1:00 AM EST on Sunday, December 28th, we identified a period of higher-than-normal latency caused by a pre-planned customer event that resulted in a significant traffic spike. Although we did not observe any service failures or timeouts, our systems experienced degraded service due to the increased latency. Systems recovered quickly once peak traffic subsided and are now operating normally. We expect performance to remain stable moving forward.

Read the full incident report →

Started Dec 16, 2025, 08:59 PM UTC· resolved Dec 16, 2025, 09:03 PM UTC

1. resolved Dec 16, 2025, 09:56 PM UTC

   Between 3:59 and 4:03 EST, customers may have experienced degraded performance when processing transactions that required 3D Secure (3DS) authentication via our Global 3DS service. Performance has since returned to normal, and no further impact has been observed. If you have any questions about potential impact to your transactions during this time window, please reach out to Spreedly Support for assistance.

2. postmortem Dec 18, 2025, 09:26 PM UTC

   # _December 18, 2025 — Delayed Processing of Some 3DS Global Transactions_ ‌ _A targeted update intended for Gateway-Specific 3DS processing impacted a subset of 3DS Global transactions. This caused a small number of transactions to remain in a "Pending" state rather than progressing to completion._ ### What Happened Between 08:59 PM and 09:03 PM UTC on December 16, 2025, a deployment intended to update Gateway-Specific 3DS logic inadvertently affected the processing of 3DS Global transactions. As a result, 52 transactions failed to reach a final state and remained "pending." During this 4-minute window, customers attempting a 3DS-authenticated operation would have encountered an interruption, preventing their transactions from completing. Our team identified the impact immediately and rolled back the change to restore normal service. ### Next Steps _We have successfully identified and corrected the underlying logic error within the change. We are also enhancing our automated deployment tests to better catch cross-module dependencies of this nature. We sincerely apologize for any disruption this caused to your operations._ ‌ — The Spreedly Team

Read the full incident report →

Started Dec 09, 2025, 03:58 PM UTC· resolved Dec 10, 2025, 08:04 PM UTC

1. investigating Dec 09, 2025, 03:58 PM UTC

   We are investigating an issue where roughly 30–40% of transactions are not appearing in Spreedly reporting since 12/03. Primary datastores and secondary services show no data loss, indicating the issue lies within the reporting ingestion process. Please reach out to Spreedly Support if you need assistance.

2. investigating Dec 10, 2025, 06:53 AM UTC

   Our team continues to investigate the root cause of the missing transaction data appearing in Spreedly reporting. All primary datastores and transactional processing are functioning normally, and there is no indication of data loss at the source. The issue appears to be isolated to the reporting ingestion process.

3. monitoring Dec 10, 2025, 02:42 PM UTC

   We have identified and fixed the issue that was causing some transaction data to be missing from Spreedly reporting. Our team is now working on backfilling the missing reporting data to ensure completeness and accuracy. All primary processing systems continue to operate normally, and there is no impact to transaction processing.

4. resolved Dec 10, 2025, 08:04 PM UTC

   Our team has completed the backfill process, and all missing transaction data has now been fully restored in Spreedly reporting. Transaction reporting is up to date and functioning as expected. All primary processing systems continued to operate normally throughout this incident, and no transaction processing impact occurred.

5. postmortem Jan 02, 2026, 09:44 PM UTC

   ### _Postmortem: January 2, 2026_ — _Transaction Reporting Degraded_ ### ‌High Level Summary Select data related to transactions did not make it into our downstream systems, causing inaccurate reports in [app.spreedly.com](http://app.spreedly.com) ### What Happened After completing system updates on December 3, 2025, approximately 30-40% of data related to transactions did not make it into our reporting tools due to a latent bug in one of our supporting applications. This caused inaccurate reports in [app.spreedly.com](http://app.spreedly.com). This did not impact any primary system functionality and no data was lost. ### Next Steps * Our team has since identified and fixed the affected application. The missing data was backfilled and subsequent data continues to flow normally * We are adding additional monitoring in order to more quickly flag discrepancies * We are continuing to make updates to simplify our data pipeline that will prevent similar issues in the future -The Spreedly Team

Read the full incident report →

Started Oct 20, 2025, 03:41 PM UTC· resolved Oct 20, 2025, 08:22 PM UTC

1. identified Oct 20, 2025, 03:41 PM UTC

   The major AWS outage on October 20, 2025, has been largely resolved for most affected services, though some residual issues may persist. Spreedly’s core platform remains stable and fully available. However, due to downstream impact from the AWS outage, some of Spreedly’s indexing endpoints and callback processing are experiencing delays. This is related to dependencies on third-party systems that were affected by the outage and are still undergoing recovery. As a result, you may continue to see intermittent latency or delayed responses for API requests involving these systems. We are actively monitoring the situation as AWS and its downstream services continue recovery efforts. For ongoing updates, please refer to the AWS status page here

2. resolved Oct 20, 2025, 08:22 PM UTC

   After closely monitoring and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected.

3. postmortem Oct 23, 2025, 06:27 PM UTC

   ### Postmortem: October 24, 2025 — _AWS outage_ Monday's AWS outage had downstream impact on some 3rd-party systems utilized by Spreedly's platform. Namely, our indexing endpoints and callbacks lagged behind due to the outage and recovery steps. ### What Happened Spreedly's 3DS global service and legacy smart routing product both faced a period of failures due to 3rd-party systems relying on AWS. ### Next Steps Spreedly will improve its response to our alerts platform, and are actively working on multi-region support slated for 2026, which should mitigate the impact of similar events going forward. -The Spreedly Team

Read the full incident report →

Started Sep 30, 2025, 06:22 PM UTC· resolved Sep 30, 2025, 07:45 PM UTC

1. investigating Sep 30, 2025, 06:22 PM UTC

   We are currently investigating a high number of 500 errors

2. investigating Sep 30, 2025, 06:25 PM UTC

   Currently we are experiencing issues with services that use payment methods

3. identified Sep 30, 2025, 06:55 PM UTC

   We are implementing a mitigation that should restore the ability to perform new tokenizations and operations on those payment methods. Some functionality, such as fingerprinting, may remain limited. Use of vaulted payment methods may remain limited. We’ll provide updates as we progress.

4. monitoring Sep 30, 2025, 07:07 PM UTC

   We have identified and resolved the issue and and monitoring for any additional impact.

5. resolved Sep 30, 2025, 07:45 PM UTC

   The incident impacting operations involving payment methods has been resolved. All services are now operating normally. During routine database maintenance, we encountered issues that affected tokenization and transactions using payment methods. We’ll share more details in the upcoming post-mortem.

6. postmortem Oct 02, 2025, 09:43 PM UTC

   ### Postmortem: _September 30, 2025_ — _Payment Method Services Inaccessible_ During a routine database configuration change, we lost connection to the Payment Methods storage table. This prevented the use of payment methods in transactions and the creation of new payment methods. ### What Happened During a routine database change to decommission extra nodes, a corruption occurred with our internal reference data. This corruption caused the database query optimizer to error when trying to process SQL queries on the payment methods storage, which resulted in the inability to access that storage table and led to errors from the application layer when trying to read, update, or create payment methods. ### Next Steps Payment methods successfully created during this period have had their fingerprints regenerated. Please contact Spreedly Support if you need the updated fingerprint values to utilize within your business processes and we will be happy to provide them. Spreedly is also working with our database service provider on a more detailed root cause analysis. Moving forward, Spreedly will be proactively engaging with our database provider when performing these types of configuration changes. -The Spreedly Team

Read the full incident report →

Started Sep 19, 2025, 03:53 PM UTC· resolved Sep 19, 2025, 05:46 PM UTC

1. investigating Sep 19, 2025, 03:53 PM UTC

   We are currently investigating elevated 3DS failures with our Global 3DS provider, Seglan.

2. resolved Sep 19, 2025, 05:46 PM UTC

   At 1:08 PM, 3DS provider, Seglan, deployed changes to address the issue. After a period of monitoring, we’ve observed normal performance and are declaring this incident resolved.

Read the full incident report →

Started Aug 19, 2025, 01:34 PM UTC· resolved Aug 19, 2025, 05:55 PM UTC

1. resolved Aug 20, 2025, 10:31 PM UTC

   On August 19th, 2025, from 9:34 a.m. to 12:55 p.m. EST, an unexpected interruption in 3DS execution occurred during a configuration change in one of our 3DS MPI providers' systems. Customers experienced an increased rate of SCA Authentication failures, resulting in the following error: "Gateway transaction not attempted due to failed SCA authentication." Our 3DS provider has since resolved the issue and is implementing stronger safeguards to prevent this from happening again. This issue is considered resolved, and no further customer impact is expected.

Read the full incident report →

Started Jun 30, 2025, 02:05 PM UTC· resolved Jun 30, 2025, 05:50 PM UTC

1. resolved Jun 30, 2025, 06:25 PM UTC

   Spreedly has identified an issue where non-unique payment method fingerprints were issued between 16:05 and 17:50 UTC. The issue has been resolved.

2. postmortem Jul 08, 2025, 02:47 PM UTC

   ## **Postmortem: June 30, 2025 — Incorrect Payment Method Fingerprints Returned** We want to sincerely apologize for an issue that occurred on **June 30, 2025**, where some of our customers received **incorrect or duplicate payment method fingerprints** when tokenizing new payment methods. This issue lasted from **4:05 PM to 5:50 PM UTC**. While no transactions failed and sensitive data was never at risk, we know that accurate fingerprinting is critical to how you manage and deduplicate payment methods, and we’re sorry for the confusion and downstream impact this may have caused. ### What Happened As part of routine security maintenance, we deployed a new encryption key used to manage payment methods. Unfortunately, the deployment included a **misconfigured Parent Data Encryption Key \(Parent DEK\)**. This misconfiguration caused our fingerprinting system to **receive an error from our encryption service**, but that error was **mistakenly interpreted as valid data**. As a result, many new payment methods received the **same fingerprint**, leading to duplicates or incorrect values being returned across all environments. Tokenization and payment processing were unaffected—transactions completed successfully, and all sensitive data remained secure. ### What We’re Doing to Prevent This We’ve already taken several steps to ensure this doesn’t happen again: * Alerting has been implemented for this step in the payment method fingerprinting service * Process changes have been implemented to prevent this type of misconfiguration in the future We’re grateful for your trust and patience. If you believe this incident may have affected your environment, or if you have any questions, please don’t hesitate to reach out to our support team. — The Spreedly Team

Read the full incident report →

Started Jun 04, 2025, 03:43 PM UTC· resolved Jun 04, 2025, 03:45 PM UTC

1. resolved Jun 04, 2025, 05:15 PM UTC

   This morning, from 11:43 AM EDT to 11:45 AM EDT, for about two minutes, we experienced a minor issue that caused intermittent 500 errors. Our team has fixed the problem, and we do not anticipate any further impact on customers.

Read the full incident report →

Started May 14, 2025, 04:07 PM UTC· resolved May 14, 2025, 04:21 PM UTC

1. investigating May 14, 2025, 04:07 PM UTC

   We are currently investigating some degradation with Google Pay transactions.

2. monitoring May 14, 2025, 04:16 PM UTC

   AFFECTED SYSTEMS OR SERVICES: Google Pay Transactions DETAILS OF THE INCIDENT: Degradation with Google Pay transactions CURRENT ACTIONS: We’ve implemented a fix and are seeing Google Pay transactions succeed. Our team is monitoring the situation closely to ensure there are no additional impacts.

3. resolved May 14, 2025, 04:21 PM UTC

   IMPACT STARTED AT: 11:18 AM EST IMPACT ENDED AT: 12:09 PM EST AFFECTED SYSTEMS OR SERVICES: Google Pay Transactions DETAILS OF THE INCIDENT: Google Pay Transactions Failing CURRENT ACTIONS: After closely monitoring Google Pay transactions and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We are completing our investigation concerning the causes of the incident and any residual impact. A post-mortem will be published. We apologize for any inconvenience or disruption.

4. postmortem May 15, 2025, 10:00 PM UTC

   # May 14, 2025 — HTTP 500 Errors and Google Pay Degradation ### ‌High-Level Summary On May 14, 2025 at 03:18 pm UTC, a configuration change was deployed to enhance application fault tolerance which, resulted in a brief period of HTTP 500 errors. While the breaking change was reverted within a minute of deployment, there were still instances of cached data which extended the impact for a total of 4 minutes of partial service outage, until 03:22 pm UTC. A secondary issue occurred as a result of the timing of this deployment coinciding with the regeneration of the Google Pay verification keys for decryption, which resulted in a 51 minutes degradation of Google Pay transactions until 04:09 pm UTC. ### What Happened A configuration change intended to enhance application fault tolerance to Redis failures inadvertently introduced serialization issues when accessing Redis keys. This resulted in new cache failing to be successfully read from the existing application instances, which resulted in HTTP 500 errors. Additionally, at the same time this change was deployed, the Google Pay verification keys were regenerated. Google Pay decryption stopped working due to the cached keys not having serialized properly. These keys then had to be deleted from the cache to enable the system to auto generate a new pair. ### Next Steps * Spreedly has implemented additional observability improvements via our automated alerting system to enhance monitoring of Google Pay services.

Read the full incident report →