Poplapay Outage History

Started Jan 02, 2025, 06:20 AM UTC· resolved Jan 02, 2025, 09:21 AM UTC

1. investigating Jan 02, 2025, 06:20 AM UTC

   We are rolling back a change and some terminals might see connectivity issues. We are sorry about any possible inconvenience.

2. monitoring Jan 02, 2025, 06:31 AM UTC

   All changes have been rolled back. Our production test devices worked continually, and most metrics were normal, but due to a partial outage of our monitoring related to the change, we could not rule out possible issues.

3. resolved Jan 02, 2025, 09:21 AM UTC

   We identified the cause of our monitoring issue, and don't believe this should have had any negative impact on payments.

Read the full incident report →

Started Dec 20, 2024, 04:12 PM UTC· resolved Dec 20, 2024, 10:12 PM UTC

1. resolved Dec 20, 2024, 04:12 PM UTC

   On December 16th, a small fraction of our customers' ECRs were unable to access our service due to DNS problems.

2. postmortem Dec 20, 2024, 04:12 PM UTC

   ## Summary On 16th of December there was an incident related to our DNS \(Domain Name Service\) records which made some merchants’ ECRs fail to connect to our systems. This problem surfaced at an unfortunate time during the height of the end-of-year shopping season, and we sincerely apologise for the problems caused by this. As a result of the now concluded analysis of the incident we are improving our monitoring and deployment processes. The root cause arises from DNS resolution breaking after an attempted move of our services to a new location. The change was rolled back, but the rollback had an unexpected side effect; the changed records were propagated to downstream DNS resolvers with a long TTL \(time to live\) even though the TTL for the changed records was set to 1 minute. ## Timeline At 06:37 EET \(GMT\+2\) we deployed a DNS change to production for the ECR API endpoint, which was rolled back automatically approximately one minute later due to a deployment failure of an unrelated \(to this incident\) resource. At 12:02 the first reports of ECR issues with some merchants came in, and at 12:07 our on-call personnel started an investigation on the incident. The cause of the problem was difficult to pin down due to the problem not being visible on any existing monitoring graphs and some red herrings \(such as suspecting problems in the terminals themselves\), but at 12:46 the potential cause for problems was identified as the production change deployed earlier in the morning. At 13:03 we deployed a fix for any affected ECRs. This fixed the problem for the affected devices within up to 18 minutes, but mostly in a few minutes due to the nature of DNS negative TTL cache expiration. ## Root cause A configuration error with the DNS NS \(name server\) record TTL caused the rollback to not fix the connectivity issues in a timely manner. We did set the TTL at 60 seconds at the parent domain’s NS record used for DNS delegation, but there was another setting for the same TTL in the subdomain’s internal NS record, which was unfortunately set at 48 hours. It is the subdomain’s TTL that takes precedence, and therefore without intervention the record would have only corrected itself after up to 2 days. Some resolvers ignore high TTL values and return e.g. with Google’s 8.8.8.8 public resolver service, a maximum TTL of 6 hours for NS records. ## Impact Any ECR device that managed to query for the NS record for the subdomain api.poplatek.com cached the new delegated name servers for 48 hours. This cache happens on the DNS resolver level, which makes all other devices using the same resolver to fail in a similar manner. This NS record was live between 06:37 and 06:38. Due to the nature of propagating modified DNS records to our name servers, it’s not possible to give more exact timestamps. Also clients cached the original A record for 60 seconds, so only a very small amount of clients managed to cache this new NS record that replaced the original A record for api.poplatek.com. Unfortunately the impact for the clients that did resolve the new NS record was catastrophic. The new name servers returned a wrong A record value for api.poplatek.com. ## Lessons learned Making changes to NS delegations simultaneously with other changes is high-risk. NS TTLs are hard to understand and to get correct, and testing environment does not always reflect production environment with regards to DNS configuration. NS TTLs are specified in two different places for the same subdomain. First, there is a TTL value with the NS records in the parent domain, but there is also a TTL for the very same NS records in the subdomain as well. We learned the hard way that it is this subdomain’s TTL that takes precedence. We thought we had set the TTL at 60 seconds but we set it in the wrong place, at the parent domain level where the DNS delegation is originally configured. We learned to always communicate changes that affect production DNS by scheduling a maintenance event at Statuspage, well before the deployment. This will help discover any issues in a more timely manner. Our monitoring is unable to detect ECR problems that affect only a small fraction of our customer base. We are going to set up new alarms which further subdivide ECR connection monitoring for anomalous drops by merchant channel. Finally, we are going to improve customer communication about the timing of the release of our public post-mortems. We understand that these incidents cause our customers uncertainty, and we want to do our best to alleviate that. We believe that these actions will help us catch these types of problems faster in the future, and improve our availability. Once again we apologize for any problems this has caused you, and wish you a successful and happy holiday season!

Read the full incident report →

Started Dec 14, 2024, 01:14 PM UTC· resolved Dec 14, 2024, 05:51 PM UTC

1. investigating Dec 14, 2024, 01:14 PM UTC

   Since around 13:25 CET we are experiencing issues with connection to acquiring, which affects payments in Sweden and Norway.

2. monitoring Dec 14, 2024, 02:07 PM UTC

   A fix was implemented and the authorizations are succeeding again from 15:00 CET onwards.

3. resolved Dec 14, 2024, 05:51 PM UTC

   This incident has been resolved.

4. postmortem Dec 23, 2024, 12:56 PM UTC

   ## Summary **Late October 2024**, we started seeing sporadic connection problems that affected a tiny number of transactions intermittently. The problems were caused by IPsec tunnel failures. We are running two tunnels from two different data centres. The problems began happening more regularly, finally escalating into a total downtime of both tunnels on a **Saturday, 14th of December**. We deeply apologise for the problems this has caused and do not consider this acceptable. Despite tuning the tunnel parameters over the past weeks, we were unable to find a solution that resolved all issues. We discovered that the tunnels sometimes failed to detect data flow interruptions promptly. We adjusted the dead peer detection \(DPD\) parameters to improve availability, which unexpectedly led to a complete downtime of both tunnels two days after the change. ## Timeline UTC\+2 * **End of October 2024**: First occurrences of a few failing transactions * **October - December**: Increasing frequency of transaction failures, though still relatively low * **Thursday, December 12th**: DPD parameters were tuned for faster detection of broken tunnels * **Saturday, December 14th**: * **13:45**: First on-call alarms about both IPsec tunnels down; expected quick resolution as before * **13:55**: Alarms closed * **14:21**: Alarms re-opened and closed again at 14:22 * **14:27**: Alarms re-opened but this time the tunnels did not self-recover * **14:38**: We requested acquirer to restart IPsec tunnels on their side * **15:14**: Statuspage incident created * **15:25**: Acquirer informed us that restarts did not help * **15:48**: Reverted DPD parameter changes and started deployment * **15:56**: First IPsec tunnel up * **16:03**: Second IPsec tunnel up, alarms closed shortly after * **16:07**: Statuspage updated with Monitoring status as transactions were succeeding * **19:51**: Statuspage updated with Resolved status as no further problems were observed * **After the weekend downtime**: Continued investigation revealed that UDP Encapsulation \(NAT-T\) might resolve all issues * **Monday, December 16th, around noon**: Deployed NAT-T configuration for the first tunnel * **Wednesday, December 18th, around noon**: Deployed NAT-T configuration for the second tunnel after observing zero downtime with the first tunnel * **Since then:** No errors observed ## Root cause The root cause of the issues remains unclear due to the complexity of IPsec debugging. There were no error messages pinpointing the cause of the tunnel failures. We have identical configuration with other acquirers without issues, and no errors in the staging environment. Through trial and error, we found that using UDP Encapsulation \(NAT-T\) resolved the issues for this particular production connection, though the exact reasons remain unknown. We recommend using mTLS instead of IPsec but that is not always available. ## Impact Purchase transactions occasionally failed, but retries were mostly successful. Unfortunately, on **December the 14th**, the problems escalated, resulting in a complete downtime of purchase transactions for all merchants using the affected acquirer from **14:27** until **16:00**. A total of 894 purchase transaction attempts failed due this problem, across all affected merchants. ## Lessons learned * **IPsec Complexity:** When problems arise, one must keep on trying to find a suitable configuration until problems are solved. Previously stable connection can fail without any configuration changes, highlighting the importance of detailed monitoring. * **Alarm Fatigue:** We must avoid alarm fatigue, where recurring issues that previously resolved without intervention are dismissed as routine. Faster reaction times could have been achieved if the issue was not considered business as usual. Additionally, we must never consider acquirer connection issues as business as usual. * **Alternative Solutions:** We advocate for alternative connection solutions to avoid IPsec, but that requires changes beyond our control. We support our partners with knowledge and experience to encourage migration to more modern solutions. We are pleased that the tunnels are now functioning as expected. We apologise for the problems and want to wish you a pleasant holiday season and a thriving business!

Read the full incident report →

Started Dec 07, 2024, 11:17 AM UTC· resolved Dec 08, 2024, 01:10 PM UTC

1. investigating Dec 07, 2024, 11:17 AM UTC

   There is a sporadic networking issue towards NDS acquiring which causes a small part of authorizations to fail. Connections to other acquirers are unaffected. Issue started 10:24 UTC

2. monitoring Dec 07, 2024, 12:07 PM UTC

   Situation has stabilized, we'll keep on monitoring it

3. resolved Dec 08, 2024, 01:10 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Nov 04, 2024, 12:02 AM UTC· resolved Nov 04, 2024, 02:35 AM UTC

1. investigating Nov 04, 2024, 12:02 AM UTC

   We are currently investigating this issue.

2. monitoring Nov 04, 2024, 02:08 AM UTC

   Nets Finland acquiring had an incident causing authorizations to decline between 1:08 and 3:45 EET. Incident has now been resolved and authorizations are working normally. We are monitoring the situation.

3. resolved Nov 04, 2024, 02:35 AM UTC

   This incident has been resolved.

Read the full incident report →