Proxyclick Outage History

Started Jul 18, 2025, 10:07 AM UTC· resolved Jul 25, 2025, 09:19 AM UTC

1. identified Jul 18, 2025, 10:07 AM UTC

   We are currently investigating an issue affecting the generation of printed QR codes in the latest production release of our Kiosk software (Version V4.1.27, released July 17, 2025). Our engineering team is actively working on a fix. In the meantime, if you have not yet updated to this version, we recommend pausing the update to avoid disruption. We sincerely apologize for the inconvenience and appreciate your patience. Further updates will be posted here as soon as they become available. Next Update: Within 24 hours

2. identified Jul 18, 2025, 10:49 AM UTC

   Workaround: We’ve confirmed that this issue impacts only unexpected (walk-in) visitors. As a temporary workaround, we recommend registering visits in advance to ensure QR code badges are generated correctly. Fix: Our engineering team has developed a fix and submitted it for expedited review with the Apple App Review Board. We’ll notify you as soon as the updated version is approved and available for download. Thank you for your continued patience. Next Update in 4 hours.

3. identified Jul 18, 2025, 10:39 PM UTC

   We are currently still awaiting final approval from Apple. In the meantime, we recommend using the following workaround: please register visitors in advance to ensure QR code badges are generated correctly. We apologize for the inconvenience and appreciate your continued patience. Restoring full functionality remains our top priority. Our next update will be provided at 2:30 AM UTC. Thank you for your understanding.

4. identified Jul 21, 2025, 08:19 AM UTC

   We’re currently in the “In Review” phase and are waiting for the next steps from Apple. In the meantime, we recommend a temporary workaround: Please pre-register visitors to ensure QR code badges are generated without issues. We understand this may cause some inconvenience and truly appreciate your patience. Resolving this issue remains our top focus, and we’re actively working toward a resolution. Our next update will be shared at 02:00 PM UTC.

5. identified Jul 21, 2025, 02:59 PM UTC

   The app is going through some changes as advised by Apple and will be submitting it again for publishing. In the meantime, we recommend a temporary workaround: Please pre-register visitors to ensure QR code badges are generated without issues. We understand this may cause some inconvenience and truly appreciate your patience. Resolving this issue remains our top focus, and we’re actively working toward a resolution. Our next update will be shared by 1330 UTC on 22 July 2025.

6. identified Jul 22, 2025, 02:31 PM UTC

   We understand the importance of this app update and are actively following up with Apple to expedite the process. We’ll provide updates on the status page as soon as we receive any further information. Thank you for your patience.

7. monitoring Jul 24, 2025, 10:35 AM UTC

   The new version of Kiosk App is available on the App Store now. Please update to latest version. With this the issue stands resolved. We shall continue to monitor this till 0900 UTC on 25 July 2025 after which it will be resolved.

8. resolved Jul 25, 2025, 09:19 AM UTC

   We are now marking this as resolved, and an RCA will be shared within the next 10 business days. Thank you for your patience.

Read the full incident report →

Started Jun 23, 2025, 04:18 PM UTC· resolved Jun 26, 2025, 08:47 AM UTC

1. investigating Jun 23, 2025, 04:18 PM UTC

   Visitors are not receiving SMS notifications in Belgium. We are currently investigating the issue with our third party vendor. Next update by 11:30 UTC on 24 June 2025.

2. investigating Jun 24, 2025, 12:24 PM UTC

   Some configuration changes were advised by the Vendor and they are being implemented which is expected to resolve the issue. We shall keep this page updated with the progress. Next Update: 11:30 UTC on 25 June 2025

3. investigating Jun 25, 2025, 12:19 PM UTC

   The measures advised by Third Party Vendor Support Team are taking longer to be implemented due to technical complexities with the implementation. We shall post an update as soon as we hear from them.

4. monitoring Jun 25, 2025, 10:47 PM UTC

   A fix has been implemented. We are moving into the Monitoring Phase while we test. We will post an update as soon as testing is complete.

5. resolved Jun 26, 2025, 08:47 AM UTC

   We are pleased to inform you that the issue affecting SMS delivery to phone numbers beginning with +32 (Belgium) has been resolved. A fix was applied on our end and following a brief monitoring period, the service is now functioning as expected. A Root Cause Analysis (RCA) will be conducted and published on our Status Page within 10 business days. Thank you for your patience and cooperation. If you have any further questions or concerns, please follow up via your existing case or submit a new support ticket.

Read the full incident report →

Started May 14, 2025, 04:41 PM UTC· resolved May 16, 2025, 07:11 AM UTC

1. investigating May 14, 2025, 04:41 PM UTC

   S2 - Issues in the download and printing of badges

2. investigating May 14, 2025, 04:54 PM UTC

   We are continuing to investigate this issue.

3. investigating May 14, 2025, 05:02 PM UTC

   We have identified that customer logos are not visible from their sites, and the badges are no longer downloading or printing because they can't find the image from the linked path. Our infrastructure team is currently investigating the issue. Next update on 7:01 PM UTC.

4. identified May 14, 2025, 05:08 PM UTC

   Our infrastructure team has identified that the issue is with the image storage, and they are trying to fix this. In the meantime, as a workaround user's should be able to replace the missing images (e.g. in badge design) by uploading a new ones.

5. identified May 14, 2025, 08:39 PM UTC

   One of the three backups has been successfully restored. We are waiting for the rest to complete the restoration.

6. identified May 14, 2025, 11:33 PM UTC

   Our Cloud Operations team continues to work on restoring data. We appreciate your patience we work through this. Our next update will be provided 2am UTC.

7. identified May 15, 2025, 02:00 AM UTC

   Our Cloud Operations team continues to work on restoring data. We appreciate your patience we work through this. Our next update will be provided at 6 AM UTC.

8. identified May 15, 2025, 05:59 AM UTC

   Two of the three backups have been successfully restored. We are currently awaiting the completion of the remaining restoration processes. Our Cloud Operations team is diligently working to restore all data. We appreciate your patience as we navigate this process. The next update will be provided at or before 9 AM UTC.

9. identified May 15, 2025, 09:05 AM UTC

   The final backup is in the final stages of being restored. We shall post an update once the restoration is completed.

10. monitoring May 15, 2025, 09:21 AM UTC

    All the backups have been restored successfully. As the issue has now been resolved, we shall monitor the situation for next 4 hours and progress it to resolved after that.

11. monitoring May 15, 2025, 01:02 PM UTC

    As the issue was widespread and affected the Americas region as well, we shall continue to monitor the situation during Americas working hours today and mark the incident as resolved at or after 0700 UTC on 16 May 2025.

12. resolved May 16, 2025, 07:11 AM UTC

    We are pleased to inform you that the issue concerning the missing downloads for Visitor has been successfully resolved following the restoration of backups. Our internal team has completed all necessary remediation steps and confirmed that the service is operating normally after a period of monitoring. A detailed Root Cause Analysis (RCA) is currently underway to thoroughly investigate the incident. The report will be published on our Status Page within the next 10 days. We sincerely appreciate your patience and cooperation throughout this process.

13. postmortem May 30, 2025, 10:43 AM UTC

    We are grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident. **Type of Event:** S2 - All Access Control Systems are down, ACS Extender showing as offline. **Services/Modules Impacted:** Access Control System **Issue Summary/Background:** All the clients using ACS reported that they are unable to scan the QR codes. **Root Cause:** Eptura CloudOps investigated the message processing mechanisms, ensuring all pending tasks were addressed. Our team discovered that incorrect timing parameters were causing the ACS service to stop unexpectedly. We took immediate action by updating the Last Fire and Next Fire event date parameters in the database, effectively restoring the ACS services. **Remediation:** These entries for the L_ast Fire_ and _Next Fire_ event dates were updated with accurate values in the database for the scheduling system, which restored normal ACS service functionality. **Timeline \(UTC\):** **03 May 2025** 20:00 UTC - Customers first reported the issue with the QR codes. **05 May 2025** 14:04 UTC - Issue was identified and reported to Eptura CloudOps/Infra team. 15:35 UTC - Eptura CloudOps team fixed the issue in the database and requested to check with the customers. 17:15 UTC - Customers confirmed the resolution to the issue. **Total Duration of Event:** 1 day and 21 hours **Preventive Actions:** Initially, our monitoring strategy relied on examining logs and the user interface \(UI\). However, during this incident, the UI incorrectly showed that the ACS service was online, which delayed our response. To prevent similar issues in the future, we are enhancing our monitoring approach. We will now focus on the table that manages scheduler operations, ensuring that all changes to relevant fields are logged. Using this data, we will set up alerts to notify our team when the Next Fire time exceeds a predetermined threshold. This proactive monitoring will enable us to respond more quickly to incidents and improve our overall operational efficiency. Thank you for your patience and support as we continue to enhance our services.

Read the full incident report →

Started May 05, 2025, 02:14 PM UTC· resolved May 06, 2025, 12:45 PM UTC

1. investigating May 05, 2025, 02:14 PM UTC

   We are currently investigating an issue with Proxyclick. We will update you when we have more information.

2. investigating May 05, 2025, 02:21 PM UTC

   All Access Control Systems are down. Our teams are actively investigating the issue and we will update you on the progress. Next Update - 16:30 UTC

3. monitoring May 05, 2025, 03:38 PM UTC

   As per the communication from our infrastructure team, the ACS platform is now online. We will continue to monitor the situation for the next 24 hours.

4. resolved May 06, 2025, 12:45 PM UTC

   We are pleased to inform you that the issue with the Access Control System for Visitor has been resolved. Our internal team has completed the necessary actions and verified that the service is functioning normally after a brief monitoring period. A Root Cause Analysis (RCA) will be conducted to understand the incident in detail. It will be available on our Status Page within 10 days. Thank you for your patience and cooperation throughout this process. If you have any further questions or concerns, please feel free to reach out.

5. postmortem May 19, 2025, 08:01 AM UTC

   We are grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident. **Type of Event:** S2 - All Access Control Systems are down, ACS Extender showing as offline **Services/Modules Impacted:** Access Control System **Issue Summary/Background:** All the clients using ACS reported that they are unable to scan the QR codes. **Root Cause:** Eptura CloudOps investigated the message processing mechanisms, ensuring all pending tasks were addressed. Our team discovered that incorrect timing parameters were causing the ACS service to stop unexpectedly. We took immediate action by updating the Last Fire and Next Fire event date parameters in the database, effectively restoring the ACS services. **Remediation:** These entries for the L_ast Fire_ and _Next Fire_ event dates were updated with accurate values in the database for the scheduling system, which restored normal ACS service functionality. **Timeline \(UTC\):** **03 May 2025** 20:00:00 UTC - Customers first reported the issue with the QR codes. **05 May 2025** 14:04:00 UTC - Issue was identified and reported to Eptura CloudOps/Infra team. 15:35:00 UTC - Eptura CloudOps team fixed the issue in the database and requested to check with the customers. 17:15:00 UTC - Customers confirmed the resolution to the issue. **Total Duration of Event:** 1 day and 21 hours **Preventive Actions:** Initially, our monitoring strategy relied on examining logs and the user interface \(UI\). However, during this incident, the UI incorrectly showed that the ACS service was online, which delayed our response. We are enhancing our monitoring approach to prevent similar issues in the future. We will now focus on the table that manages scheduler operations, ensuring that all changes to relevant fields are logged. Using this data, we will set up alerts to notify our team when the Next Fire time exceeds a predetermined threshold. This proactive monitoring will enable us to respond more quickly to incidents and improve our overall operational efficiency. Thank you for your patience and support as we continue to enhance our services.

Read the full incident report →

Started Feb 11, 2025, 05:32 PM UTC· resolved Feb 12, 2025, 04:41 AM UTC

1. resolved Feb 11, 2025, 11:30 PM UTC

   Users were getting Service Unavailable error while trying to access Eptura Visitor via SSO. The issue was intermittent in nature.

2. postmortem Apr 07, 2025, 03:12 PM UTC

   We are grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident. ‌ **Type of Event:** S2 - Visitor - Intermittent SSO Service Error **Services/Modules Impacted:** Visitor application login \(intermittent issue\) **Root Cause:** One node of SAML connection was not responding and this caused intermittent issue for SSO authentication. **Remediation:** The team investigated the issue and identified that there were errors observed in the logs. To resolve the issue, both the notes were restarted. The team monitored application after the restart to ensure that no further errors were logged. ‌ **Timeline:** _All times listed in UTC_ **11 Feb 17:32** - Issue Reported to Eptura Support. **12 Feb 02:41** - The support team identified that the issue was intermittent and was caused by Infra related item. **12 Feb 04:41** - Both the nodes were restarted by Eptura CloudOps team which resolved the issue. **Total Duration of Event:** 11 hours 8 minutes **Preventive Action:** Our focus is on enhancing monitoring, refining management processes, and elevating team training. To mitigate future occurrences, the following measures have been implemented by our CloudOps team: * Monitoring of [saml.proxyclick.com](http://saml.proxyclick.com/). * Regularly monitoring the page internally that reports errors to the server.

Read the full incident report →

Started Nov 18, 2024, 01:51 PM UTC· resolved Nov 18, 2024, 04:22 PM UTC

1. investigating Nov 18, 2024, 01:51 PM UTC

   Our Infra team is currently investigating reports of increased loading times and slow response in the Proxyclick admin dashboard.

2. investigating Nov 18, 2024, 02:45 PM UTC

   We are continuing to investigate this issue.

3. resolved Nov 18, 2024, 04:22 PM UTC

   We are pleased to share that this incident is resolved as we have confirmed full restoration of admin response time performance. We will publish our root cause analysis findings on this incident within 10 business days. Thank you for your continued patience.

4. postmortem May 19, 2025, 08:03 AM UTC

   We are grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident. **Type of Event:** S2 – Visitor \(PXC\) API and Dashboard Performance Issues **Services/Modules Impacted:** Visitor Dashboard, web application, and API **Issue Summary/Background:** An increase in API response time and a significant spike in HTTP 500 errors were observed. Upon investigation, the logs revealed a rise in MySQL deadlocks and application server errors. Specifically, the following error was noted in the application logs: _\[HPM\] Error occurred while trying to proxy request /full/app/CO-DEDP117/vm/kiosk/instances from_ [_app.example.com_](http://app.example.com/) _to_ [_https://api.example.com_](https://api.example.com/) _\(ECONNRESET\) \(_[Errors | Node.js v23.11.0 Documentation](https://nodejs.org/api/errors.html#errors_common_system_errors)) ‌ These issues affected the overall application performance and user experience. **Root Cause:** High latency and increased MySQL deadlocks. **Remediation:** While reviewing MySQL and application server logs, the team identified an increase in deadlocks in MySQL logs and application connection resets. To mitigate the issue, the Eptura CloudOps team performed a database. The restart helped in returning the application functionality to normal. We also noticed that API response times significantly improved and the HTTP 500 errors decreased substantially. **Timeline:** _All times listed in CEST_ _9:22 a.m: A large spike in 5xx HTTP response errors_ and _elevated response times from API endpoints were observed._ 2:56  _p.m.: An incident was reported, prompting Eptura to initiate the investigation._ _3:51 p.m: Eptura updated the Visitor status page to reflect the incident and investigation._ _4:57 p.m.: The Eptura Infra team identified the root cause of the issue and suggested that the response times have now been stable._ _5:53 p.m: The Eptura team updated cases to ask customers for initial feedback on the issue._ _6:22 p.m: The Eptura team updated the status page confirming the issue is resolved._ **Total Duration of Event:** 1 Hour 12 minutes **Preventive Actions**: To mitigate future occurrences, we have scheduled a proactive MySQL service restart every Monday before working hours to maintain database stability.

Read the full incident report →