Spreedly Outage History

Started Apr 14, 2025, 07:55 PM UTC· resolved Apr 14, 2025, 08:33 PM UTC

1. investigating Apr 14, 2025, 07:55 PM UTC

   Spreedly has detected an issue that may impact customers, including potentially resulting in failed transactions. While we are providing an early notification in an effort to alert you as quickly as possible, we are still investigating the actual scope and impact and will provide an update as soon as more details are available.

2. investigating Apr 14, 2025, 08:08 PM UTC

   We are continuing to investigate this issue.

3. identified Apr 14, 2025, 08:13 PM UTC

   The issue has been identified and a fix is being implemented.

4. monitoring Apr 14, 2025, 08:18 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Apr 14, 2025, 08:33 PM UTC

   This incident has been resolved.

6. postmortem Apr 28, 2025, 11:37 PM UTC

   # **April 14, 2025 — Certificate Issue Impacts Core Requests** ‌ On April 14 at 3:47 PM ET, a platform-wide incident occurred that caused all core Spreedly requests to fail. The issue was identified and declared at 3:49 PM, mitigated by 4:13 PM, and fully resolved by 4:32 PM. Total impact duration was 26 minutes. During this window, merchants were unable to complete any transactions through the Spreedly platform due to system-wide connectivity failures. ## What Happened An intermediate certificate on our core service hosts expired April 14th at 3:47pm EST, which broke the certificate chain, preventing HTTP requests between Spreedly internal services and subsequently led to responses of HTTP 500 errors for all requests. ## Next Steps * Spreedly is completing an audit of all certificate chains to identify potential risks and update our catalog of expiration dates. * Spreedly has a plan in place to migrate the management of all certificate chains to our cloud provider.

Read the full incident report →

Started Apr 02, 2025, 08:54 AM UTC· resolved Apr 02, 2025, 10:49 AM UTC

1. investigating Apr 02, 2025, 08:54 AM UTC

   We are currently investigating an issue where the stylesheet express-mobile-3.1.0.min.css is returning a 404 Not Found error, causing the user interface to break in the Spreedly Express integration.

2. monitoring Apr 02, 2025, 10:24 AM UTC

   The issue with the missing stylesheet express-mobile-3.1.0.min.css causing the user interface to break in the Spreedly Express integration has been resolved. We are now actively monitoring the system to ensure the issue does not recur.

3. resolved Apr 02, 2025, 10:49 AM UTC

   The issue has been fully resolved, and the system is functioning as expected. No further incidents have been observed.

Read the full incident report →

Started Mar 15, 2025, 06:50 AM UTC· resolved Mar 15, 2025, 01:40 PM UTC

1. investigating Mar 15, 2025, 06:50 AM UTC

   We are investigating an elevated rate of 5xx errors affecting Spreedly core services starting at 01:36AM EST. While we are providing an early notification in an effort to alert you as quickly as possible, we are still investigating the actual scope and impact and will provide an update as soon as more details are available. Our team is actively working to identify the root cause and mitigate the impact. We will provide further updates as soon as we have more information. Thank you for your patience.

2. monitoring Mar 15, 2025, 07:32 AM UTC

   As of 3:10 AM EST, our system availability has been fully restored and we are seeing stability return. We have implemented a fix and stabilized Spreedly Core Services. Our team is closely monitoring the situation to ensure no further impact.

3. monitoring Mar 15, 2025, 11:35 AM UTC

   As of 3:10 AM EST, our system availability has been fully restored and we are seeing stability return. We have implemented a fix and stabilized Spreedly Core Services. Our team is closely monitoring the situation to ensure no further impact.

4. resolved Mar 15, 2025, 01:40 PM UTC

   IMPACT STARTED AT: 2:12 AM EST IMPACT ENDED AT: 2:40 AM EST After closely monitoring Spreedly Core Services and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We are completing our investigation concerning the causes of the incident and any residual impact. A post-mortem will be published. We apologize for any inconvenience or disruption.

5. postmortem Mar 21, 2025, 02:07 AM UTC

   # March 15th, 2025 — Payment Method Storage Errors _On March 15th, 2025 at 06:12 UTC Spreedly systems encountered a system resource constraint which led to intermittent API call failures resulting in_ `404` _and_ `500` errors being returned to customers. _The unresponsive application containers were restarted allowing the system to return to normal at 06:47 UTC_ ## What Happened On 2025-03-15 between 06:12 and 06:47 UTC, calls to Spreedly’s payment method tokenization endpoints intermittently failed resulting in `HTTP 500` errors. Intermittent elevated rates of `HTTP 404` errors when attempting to use a payment method to transact were also encountered. These failures were the result of internal system resource limitations. ## Next Steps * Spreedly has implemented additional resource monitoring in our automated alerting system to prevent future overruns. * System resources have been adjusted to prevent similar situations in the future.

Read the full incident report →

Started Feb 16, 2025, 09:43 AM UTC· resolved Feb 16, 2025, 10:39 AM UTC

1. investigating Feb 16, 2025, 09:43 AM UTC

   We are investigating an elevated rate of 5xx errors affecting Spreedly core services. This issue began at 3:56 AM EST. Our team is actively working to identify the root cause and mitigate the impact. We will provide further updates as soon as we have more information. If you have any questions or concerns, please reach out and our support team will be happy to assist.

2. monitoring Feb 16, 2025, 10:16 AM UTC

   As of 4:44 AM EST, our systems have been resourcing this issue, and we are seeing stability return. We have implemented a fix and stabilized Spreedly Core Services. Our team is closely monitoring the situation to ensure no further impact.

3. resolved Feb 16, 2025, 10:39 AM UTC

   After closely monitoring Spreedly Core Services and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We are completing our investigation concerning the root cause of the incident and any residual impact. A post-mortem will be published. We apologize for any inconvenience or disruption.

4. postmortem Feb 20, 2025, 10:16 PM UTC

   # February 16, 2025 — Intermittent API Failures Resulting in 404 and 500 errors ‌ _On February 16, 2025 at 08:55 UTC Spreedly systems encountered a system resource constraint which led to intermittent API call failures resulting in_ `404` _and_ `500` errors being returned to customers. _The unresponsive application containers were restarted allowing the system to return to normal at 09:55 UTC._ ### What Happened On 2025-02-16 between 08:55 and 09:55 UTC calls to Spreedly’s payment method tokenization endpoints intermittently failed resulting in `HTTP 500` errors. Intermittent elevated rates of `HTTP 404` errors when attempting to use a payment method to transact were also encountered. These failures were the result of internal system resource limitations. ### Next Steps * Spreedly has implemented additional resource monitoring in our automated alerting system to prevent future overruns. * System resources have been adjusted to slow similar situations in the future.

Read the full incident report →

Started Dec 12, 2024, 06:04 PM UTC· resolved Dec 12, 2024, 07:25 PM UTC

1. investigating Dec 12, 2024, 06:04 PM UTC

   Spreedly has discovered an issue that could potentially impact new payment method creation. We are taking all necessary measures to investigate the matter and determine the extent of the situation. We will provide an update as soon as more details are available.

2. monitoring Dec 12, 2024, 06:32 PM UTC

   From 12:47 to 12:58 PM EST, we encountered an issue that affected customers attempting to tokenize a payment method and transacting with a payment method, resulting in a 500 error. We have identified the problem and implemented a fix. Our team is closely monitoring the situation to ensure there are no further impacts. If you need additional assistance, please reach out to our Support team.

3. resolved Dec 12, 2024, 07:25 PM UTC

   After closely monitoring and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We apologize for any inconvenience or disruption.

4. postmortem Dec 17, 2024, 10:59 PM UTC

   # December 12th, 2024 — Payment Method Tokenization Errors Server maintenance to reduce log volume caused an internal service to become unavailable, causing payment method tokenization requests to fail. ## What Happened At approximately 17:45 UTC on December 12th, 2024, a logging configuration change made to reduce log volume resulted in an internal service becoming unavailable. Traffic was shifted to an alternate service during the outage window to reduce impact. At approximately 18:01 the internal service resumed normal operation and service was restored. Payment transactions were negatively impacted during this time. ## Next Steps Spreedly Engineering is investigating the coupling of the configuration change to the internal service and will be taking steps to prevent the issue from occuring in the future.

Read the full incident report →

Started Dec 12, 2024, 12:42 AM UTC· resolved Dec 12, 2024, 02:05 AM UTC

1. investigating Dec 12, 2024, 12:42 AM UTC

   Spreedly is investigating an issue that could potentially impact our services, leading to 404 errors on the Global Collect gateway. We are taking all necessary measures to investigate the matter and determine the extent of the situation. We will provide an update as soon as more details are available. Thank you for your patience.

2. resolved Dec 12, 2024, 02:05 AM UTC

   We have identified an issue with a code change that was deployed around 3:35 pm EST that affected the Global Collect gateway. We have rolled back this code change and are now seeing successful transactions. After a period of monitoring, we have not seen any recurrence of this issue. The incident is being considered resolved. We will publish a postmortem on our status page as soon as possible. We apologize for any inconvenience this has caused.

3. postmortem Dec 16, 2024, 07:04 PM UTC

   # December 11th, 2024 — Global Collect Gateway Transaction Failures A change to the Global Collect gateway integration resulted in transaction failures for a subset of customers. ## What Happened At 20:50 UTC, an update was made to the Global Collect gateway integration. Initial monitoring and observability did not surface issues. At 23:53 UTC, Spreedly Engineering began researching an elevated number of transaction failures on the Global Collect gateway. After investigation, it was determined that the prior update to the Global Collect gateway was causing the increase in transaction failures. At 01:02 UTC on December 12th, 2024, the change was reverted and service to the Global Collect gateway was restored. ## Next Steps Spreedly Engineering is reviewing and improving internal observability to better surface gateway-specific transaction failure rate anomalies.

Read the full incident report →

Started Dec 11, 2024, 07:15 PM UTC· resolved Dec 11, 2024, 07:41 PM UTC

1. resolved Dec 11, 2024, 08:50 PM UTC

   While maintaining a backend service to increase capacity, we saw intermittent errors during the timeframes of 2:28 PM EST and 2:30 PM EST and again during the timeframes of 2:39 PM EST and 2:41 PM EST. Our internal monitors picked up this issue, and we resolved it quickly. We have since seen the system return to normal, and we do not expect any further customer impact. This incident is considered resolved. We apologize for any inconvenience this may have caused.

2. postmortem Dec 16, 2024, 07:16 PM UTC

   # December 11th, 2024 — Internal System Errors Server maintenance performed during Spreedly’s regular change window to increase capacity to an internal service caused two brief periods of service interruption to APIs that access decrypted information. ## What Happened At approximately 19:15 UTC, Spreedly Engineering initiated a capacity expansion of an internal service used for data decryption. From 19:28 to 19:30, a first wave of failed API requests during the maintenance window. A second wave of failed API requests occurred from 19:39 to 19:41 UTC when rebalancing request traffic to the capacity-expanded internal service. Service was restored at 19:41 UTC, and the system was fully operational. API calls that required decrypted data were impacted during the outage timeframes. ## Next Steps Spreedly Engineering is improving internal observability, implementing automated monitors, and investigating the use of automation for scaling capacity in the future to prevent this issue from recurring.

Read the full incident report →

Started Dec 10, 2024, 04:59 PM UTC· resolved Dec 10, 2024, 08:22 PM UTC

1. identified Dec 10, 2024, 04:59 PM UTC

   Spreedly has discovered an issue that could potentially impact our services, leading to errors with caching the CVV. We are taking all necessary measures to investigate the matter and determine the extent of the situation. We provide an update as soon as more details are available. Thank you for your patience.

2. monitoring Dec 10, 2024, 05:16 PM UTC

   Spreedly experienced an issue with Caching the CVV. Once identified, we immediately began working on a resolution. We’ve implemented a fix and stabilized the system. Our team is monitoring the situation closely to ensure there are no additional impacts. Please reach out to our Support team if you need any additional help.

3. resolved Dec 10, 2024, 08:22 PM UTC

   After closely monitoring and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We are completing our investigation with regard to the causes of the incident and any residual impact. A post-mortem review will be published. We apologize for any inconvenience or disruption.

4. postmortem Dec 12, 2024, 10:32 PM UTC

   ## December 10th, 2024 — CVV Caching Issue At approximately 15:36 UTC on December 10th, 2024, Spreedly Engineering attempted a code change to an internal service that inadvertently prevented Spreedly from storing CVVs for payment methods when a /recache request was made. The adverse change resulted in a partial outage of Spreedly’s transactional API between 15:36 UTC and 17:07 UTC. Service was fully restored at 17:07 UTC. After a period of monitoring, Spreedly’s systems are stable, and no further impact is expected on customers. ## What Happened At approximately 15:36 UTC on December 10th, 2024, Spreedly Engineering attempted a code change to an internal service that inadvertently prevented Spreedly from extending the time a CVV exists in the cache. The adverse change resulted in a subset of transactions failing for customers who rely on Spreedly’s CVV recaching mechanism to complete a transaction. Spreedly Engineering team quickly mobilized and identified the adverse code deployment as the issue and immediately began reverting the change. The code reversion was completed at approximately 17:07 UTC, with customers reporting no additional impact after the code reversion. ## Next Steps Spreedly Engineering is reviewing internal observability to determine how to automate the detection of such an issue in the future.

Read the full incident report →

Started Dec 03, 2024, 04:05 PM UTC· resolved Dec 03, 2024, 06:22 PM UTC

1. monitoring Dec 03, 2024, 04:05 PM UTC

   We experienced a brief period of system instability during which multiple endpoints were impacted. This issue resulted in intermittent errors and disruptions in service. Starting at 3:15 PM UTC our team promptly identified the root cause of the issue, and as of 3:20 PM UTC, the problem was resolved. Since then, we have not observed any recurrence of this issue. We are closely monitoring the system to ensure continued stability and are taking steps to prevent similar incidents in the future. If you have any questions or concerns, please reach out and our support team will be happy to assist.

2. resolved Dec 03, 2024, 06:22 PM UTC

   After closely monitoring and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected. We are completing our investigation with regard to the causes of the incident and any residual impact. We apologize for any inconvenience or disruption.

Read the full incident report →

Started Oct 28, 2024, 11:44 AM UTC· resolved Oct 28, 2024, 01:16 PM UTC

1. investigating Oct 28, 2024, 11:44 AM UTC

   We are investigating the potential degradation in 3DS2 processing, with specific focus on transactions not progressing beyond the pending state.

2. resolved Oct 28, 2024, 01:16 PM UTC

   After investigation, we have determined that there are no systemic issues with 3DS processing. Properly configured 3DS transactions are completing as expected. Thank you for your understanding.

Read the full incident report →