Vusion Outage History

Started Sep 15, 2026, 01:47 PM UTC· resolved Sep 15, 2026, 09:29 PM UTC

1. identified Sep 15, 2026, 01:47 PM UTC

   We identified a major incident affecting one data processing server in Europe following a configuration change deployed on September 14 at 09:15 UTC. An incorrect configuration introduced during this change caused unexpected data processing behavior, resulting in data corruption for part of the incoming data. The issue was corrected at **06:56 UTC**. New incoming data has since been processed correctly. Recovery of the affected data started at **08:18 UTC**. As part of the recovery process: * All data received since the configuration change was replayed in chronological order to preserve data consistency. * Corrupted data was identified and cleaned up before replay. Recovery is still ongoing and has been completed for the vast majority of affected data. We are currently investigating a limited number of remaining cases where the replay may not have completed successfully. We will provide a root cause analysis and preventive actions once the investigation is complete. We apologize for the disruption and appreciate your patience.

2. monitoring Sep 15, 2026, 07:09 PM UTC

   Our engineers have deployed a fix and confirmed that the issue has been corrected. The issue was corrected at **06:56 UTC**. New incoming data has since been processed correctly. All data received since the configuration change is being replayed in chronological order to preserve data consistency. We will provide a root cause analysis and preventive actions once the investigation is complete. We apologize for the disruption and appreciate your patience.

3. identified Sep 15, 2026, 07:32 PM UTC

   We identified an issue in the recovery process that prevented some data from being replayed correctly. The issue has now been identified and corrected, and the recovery process has resumed. The affected data is currently being replayed in a controlled manner. Based on the current processing rate, we estimate that the recovery will take approximately 6 hours to complete. We continue to closely monitor the recovery and will provide further updates on its progress.

4. resolved Sep 15, 2026, 09:29 PM UTC

   This incident has been resolved, data is being processed as expected. If doubt, please reach out to your support. We will provide a root cause analysis and preventive actions once the investigation is complete. We apologize for the disruption and appreciate your patience.

5. postmortem Sep 30, 2026, 07:58 AM UTC

   # RCA — Data Integration Incident ### Issue On September 14, a configuration change intended to handle specific special characters during data integration caused some item identifiers to be incorrectly modified. The incident affected approximately 9% of our European installed base. No source data was permanently lost, and the original data required for recovery remained available. ## Analysis The configuration was intended to remove a limited set of special Unicode characters from incoming identifiers. However, the configuration expression was interpreted differently from its intended behavior and caused legitimate individual characters to be removed from some identifiers during integration. As a result, some incoming items were processed with modified identifiers. In certain cases, this also created incorrect associations between item identifiers and their references, which could subsequently affect reference-based label matching. The issue was reported by Support on September 15 at 04:30 UTC and identified at approximately 06:40 UTC. The configuration was corrected, and normal processing of new incoming data was restored by 07:09 UTC on September 15. Recovery of data received during the affected period was then initiated using an existing replay mechanism. During verification, we identified a separate issue in this mechanism that prevented the first recovery attempt from restoring the affected data correctly. The replay mechanism was corrected, and a new controlled recovery was initiated using the original source data. ## Incident Resolution and Recovery The incorrect configuration was removed and normal data integration was restored. Affected data was then cleaned and replayed from the original source data while preserving its chronological order. The final recovery started at 18:30 UTC on September 15 and completed at 02:50 UTC on September 16. The primary item data was recovered from the original source data with no permanent source-data loss. Subsequent verification identified secondary effects on some reference-based associations. A dedicated analysis and remediation process was therefore initiated to identify and correct these inconsistencies. The affected label associations identified through this analysis were corrected. Further verification has since identified that a limited number of residual item/reference inconsistencies may remain in some environments. Additional remediation is ongoing to identify and correct these residual cases. The platform has been processing new incoming data normally since the original configuration was corrected. ## Trigger Cause A configuration change intended to ignore specific special characters during data integration resulted in certain valid alphanumeric characters contained in item identifiers being removed during integration. ## Root Cause The configuration used to handle special characters was interpreted differently from its intended behavior, resulting in unintended modification of item identifiers. The configuration change was tested in pre-production, but the validation did not identify its potential impact on valid item identifiers. Post-deployment monitoring also did not detect the abnormal identifier transformations early enough. A separate defect in the existing replay mechanism complicated the recovery by preventing the initial replay attempt from successfully restoring the affected data. The incident also highlighted that our recovery procedures did not sufficiently account for secondary effects caused by incorrect identifiers, particularly reference-based associations and matching state. These effects required additional analysis after the primary data recovery had completed. ## Corrective and Preventive Actions Immediate corrective actions included correcting the configuration, fixing the replay mechanism, implementing additional recovery monitoring, replaying the affected source data, and correcting the reference-based label associations identified during verification. Following the post-incident review, we are: * Strengthening the review and validation process for production configuration changes, including independent post-deployment monitoring. * Adding safeguards to ensure that valid characters in item identifiers cannot be unintentionally modified. * Strengthening recovery procedures with explicit end-to-end validation of item data, references, and resulting associations. * Improving recovery monitoring so that unsuccessful or incomplete replay operations are detected earlier. * Extending post-recovery verification to identify residual inconsistencies that may not be visible during the primary recovery. * Reviewing recovery strategies for data-corruption incidents to ensure that restoration and deterministic replay options are evaluated early in the incident response. ## Residual Impact and Support The main recovery activities have been completed and the platform is operating normally. However, due to the nature and scale of the original data corruption, a small number of residual inconsistencies may still be discovered during ongoing verification or normal operation. We are continuing our analysis and remediation efforts to identify and correct such cases. If customers observe unexpected item associations, incorrect label content, or other behavior that may be related to this incident, we ask them to contact Vusion Support with the affected store and item or label details. These cases will be investigated and addressed as part of the ongoing remediation. We remain committed to resolving any residual effects identified from this incident.

Read the full incident report →

Started Jul 01, 2026, 08:29 AM UTC· resolved Jul 01, 2026, 04:01 PM UTC

1. identified Jul 01, 2026, 08:29 AM UTC

   Dear VusionCloud users We are currently experiencing a service disruption caused by an ongoing Microsoft Azure incident affecting the infrastructure hosting the **apa-eu-2.vusion.io** endpoint. **Impact** Customers using the **apa-eu-2.vusion.io** endpoint may experience delays in data transmission and processing. **Important:** No data is being lost. Messages and events are being queued and will be processed once the Azure service is fully restored. Our teams are actively monitoring the situation and the incident at Microsoft. We will provide updates as soon as more information becomes available or the incident is resolved. We apologize for the inconvenience and appreciate your patience.

2. monitoring Jul 01, 2026, 02:36 PM UTC

   Microsoft has mitigated the Azure incident affecting the infrastructure behind the apa-eu-2.vusion.io endpoint. Customers may still experience residual delays in data transmission while the service fully recovers. No data has been lost. Any delayed messages and events remain queued and will continue to be processed. We are monitoring the situation closely and will provide a further update once the service is fully stable.

3. resolved Jul 01, 2026, 04:01 PM UTC

   The Microsoft Azure incident affecting the infrastructure behind the **apa-eu-2.vusion.io** endpoint has been resolved. Services have returned to normal operation, and any delayed messages have been processed. No data was lost during the incident. We will continue to monitor the platform to ensure service stability.

Read the full incident report →

Started May 11, 2026, 08:22 AM UTC· resolved May 11, 2026, 02:09 PM UTC

1. investigating May 11, 2026, 08:22 AM UTC

   Dear VusionCloud users, We are currently facing slowness and delays in the user interface, especially Vusion Manager. We are also observing an increase in API errors. Our engineers are actively investigating the problem and monitoring the situation. We will update with the progress. Thank you for your understanding.

2. identified May 11, 2026, 09:43 AM UTC

   The issue has been identified in one of our database used by the UI. Backend operations such as matching and integration were not impacted. Web UI and some API calls such as GET /events, GET /timelines or POST /search could experience slowness, timeouts or occasional errors. We have taken the necessary steps to fix the problem, and the situation should have improved greatly already.

3. monitoring May 11, 2026, 09:57 AM UTC

   All delayed events have been processed and the situation should be back to normal. We will keep monitoring the situation in the coming hours.

4. resolved May 11, 2026, 02:09 PM UTC

   The incident is now resolved. The service has been stable for several hours under close monitoring, with no recurrence observed. We are confident the situation is fully mitigated. Monitoring has now returned to normal operating conditions. Thank you for your support.

Read the full incident report →

Started May 08, 2026, 08:21 PM UTC· resolved May 09, 2026, 07:07 PM UTC

1. identified May 08, 2026, 08:21 PM UTC

   Dear customers, This message concerns all customers running SubOne Access-Points. Other radio technologies are not impacted. We would like to inform you of an on-going situation currently affecting some SubOne network equipments. Our records indicate that stores equipped with a secondary SubOne access point (AP) may experience connectivity issues. This is due to an obsolete component in the current software version, which can cause the secondary APs to go offline. Please note that stores operating with a single SubOne access point are not impacted. A corrected software version is now available and resolves this issue. To restore full functionality, we kindly ask you to: - Download the package from http://download.findbox.de/patches/fbos_4.4.0b1.fbupd - Update the software on your main access point - Update the software on any secondary access points Applying the update to both devices is necessary to ensure proper operation and prevent further disruption. Our engineers are working to remotely deploy the package whenever possible. Please ensure network to *.findbox.de is allowed to allow updates. In any case, you are advised to apply packages to ensure faster recovery. If you need assistance with the update process or have any questions, our support team is available to help. We appreciate your prompt attention to this matter and we would like to offer our apologizes for any inconvenience caused. We understand that such issues have great impact to your store operations. Kind regards, Support Team

2. identified May 09, 2026, 07:07 PM UTC

   Dear customers, A campaign has been initiated overnight and is well underway to update all SubOne APs that are connected to our monitoring servers. Access points that are still running the 4.3 or lower will be updated as they connect to our monitoring servers, or upon request. You are also welcome to update them yourselves whenever possible. On behalf of Vusion, we would like to thank you for your support and understanding during this situation.

3. resolved May 09, 2026, 07:07 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 30, 2026, 10:00 AM UTC· resolved Apr 30, 2026, 10:30 AM UTC

1. resolved May 27, 2026, 10:03 AM UTC

   On April 30, 2026 around 10:00 UTC, our Europe region experienced a degradation of our API services. Requests coming from customers were reaching our platform, but part of the backend responsible for search operations in Europe stopped responding correctly. Two of the five service instances were still accepting requests but were no longer answering in a timely way. As a result, requests remained “stuck” until they hit technical timeouts. This behavior created congestion on the upstream components that route and secure traffic (API gateway and web application firewall). As connections accumulated and were not released, the network resources became saturated and new requests started to fail or time out. Traffic progressively recovered after we restarted the affected components and the system drained the accumulated requests. The situation went back to normal at 10:30 UTC What was the impact? For customers using our services in Europe during the incident window: - Slower responses and timeouts for API calls, especially search-related operations. - Higher rate of failed or abandoned requests, with some calls never receiving a response within a reasonable time. - Degraded user experience in applications relying on our APIs (delays, retries, and occasional error messages). No data loss was observed; the issue was limited to service availability and latency.

2. postmortem May 27, 2026, 10:04 AM UTC

   ## What happened? On **April 30, 2026 around 10:00 UTC**, our Europe region experienced a degradation of our API services. Requests coming from customers were reaching our platform, but part of the backend responsible for search operations in Europe stopped responding correctly. Two of the five service instances were still accepting requests but were no longer answering in a timely way. As a result, requests remained “stuck” until they hit technical timeouts. This behavior created congestion on the upstream components that route and secure traffic \(API gateway and web application firewall\). As connections accumulated and were not released, the network resources became saturated and new requests started to **fail or time out**. Traffic progressively recovered after we restarted the affected components and the system drained the accumulated requests. The situation went back to normal at **10:30 UTC** ## What was the impact? For customers using our services in Europe during the incident window: * **Slower responses and timeouts** for API calls, especially search-related operations. * **Higher rate of failed or abandoned requests**, with some calls never receiving a response within a reasonable time. * **Degraded user experience** in applications relying on our APIs \(delays, retries, and occasional error messages\). * No data loss was observed; the issue was limited to **service availability and latency**. ## What went wrong, and why? The incident was caused by a combination of a local failure and insufficient protective mechanisms: * Two backend service instances continued to accept requests but **stopped processing them correctly**, keeping connections open until long technical timeouts expired. * These long‑lived connections caused **upstream congestion** in our API gateway and web application firewall, which rely on a shared pool of network resources. * As these resources became exhausted, **new requests could no longer be served reliably**, leading to timeouts and errors for customers. ## How did we respond? Once we detected abnormal error rates and latency in Europe, we: * Identified the two unhealthy search service instances and **restarted them** to immediately restore normal behavior. * Closely monitored API success rates, latency, and network resource usage to confirm that services were recovering and remaining stable. * Reviewed detailed logs and metrics across the Europe API gateway and search services to **reconstruct the sequence of events** and validate the technical root cause. ## How are we preventing this in the future? To prevent similar issues and reduce their impact if they occur, we are implementing the following improvements: * **Stricter timeouts and fail‑fast behavior** Shorten the maximum time a request can stay open in the backend and upstream components, so unhealthy instances are detected quickly and do not block resources for long periods. * **Improved health checks and automatic recovery** Strengthen the health probes on backend instances so that services that stop responding correctly are **automatically removed from traffic and restarted**, instead of remaining partially available. * **Enhanced monitoring and alerting** Add more precise alerts on **error codes, long‑running requests, and network resource usage** in Europe, so that similar patterns are detected earlier and resolved before they become visible to customers. These changes are designed to make our services more resilient and to ensure that, if a component becomes unhealthy, it is isolated quickly and does not impact the overall experience.

Read the full incident report →

Started Apr 29, 2026, 10:11 AM UTC· resolved Apr 29, 2026, 12:36 PM UTC

1. investigating Apr 29, 2026, 10:11 AM UTC

   Dear VusionCloud users, We are experiencing high resource usage and scalability issues. We are currently investigating hte issue. Users may experience Gateway Timeout errors, or slowness in the user interface. No specific action is required on the users' side. Our engineers are actively working on this issue. THank you for your understanding.

2. identified Apr 29, 2026, 10:46 AM UTC

   The issue has been identified as a failed automatic scaling. Resources have been manually scaled out to handle the current load.

3. monitoring Apr 29, 2026, 10:49 AM UTC

   Manual scaling has been successfully performed. Failed operations are being replayed, and performance is back to expected levels. We will be actively monitoring the situation in the coming hours.

4. resolved Apr 29, 2026, 12:36 PM UTC

   All activity is now back to normal. All failed operations have been successfully replayed, and the platform is now fully available again. Thank you for your patience and understanding.

Read the full incident report →

Started Feb 25, 2026, 07:38 PM UTC· resolved Feb 25, 2026, 07:38 PM UTC

1. resolved Feb 25, 2026, 07:38 PM UTC

   Dear users, As previously communicated, we have improved our quota-control measures to ensure system stability, maintain performance levels, and provide a consistent experience across all users. We are still observing cases where customers are being blocked due to exceeding their allocated quotas. In most instances, this is occurring because usage levels were not adjusted in line with the communicated limits. We understand that service interruptions can be disruptive, and we want to emphasize that these controls were introduced as a necessary safeguard. To prevent further blocking, we encourage customers to closely monitor their consumption and “out of quota” responses in API calls. These responses are a clear indicator that the allocated limits have been reached and should trigger appropriate follow-up actions, whether that is customer communication, quota adjustment, or usage optimization. Would you be blocked and require support any required quota reviews or capacity increases, please reach out to your support who will assist. We thank you for your understanding and remain available, would you need assistance. Please refer to: https://ses-esl.atlassian.net/wiki/spaces/VRN/pages/5780013057/January+2026+-+VUSION+Cloud+Public+release https://ses-esl.atlassian.net/wiki/spaces/VRN/pages/5907283969/February+2026+-+VUSION+Cloud+Public+release https://ses-esl.atlassian.net/wiki/spaces/VRN/pages/4232839169/API+Best+Practices

Read the full incident report →

Started Jan 23, 2026, 10:01 AM UTC· resolved Jan 23, 2026, 01:22 PM UTC

1. identified Jan 23, 2026, 10:01 AM UTC

   Dear customers On January 23th, 2026, at 4:48 AM UTC, a subset of customers experienced a disruption to their Access Points (APs), which intermittently lost connection to VUSION Cloud. Issue is primarly linked to a certificate issue on VTransmit 001t005. Our cloud engineers are working on a workaround and are expecting a solution shortly We would like to thank you for your understanding.

2. monitoring Jan 23, 2026, 10:14 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jan 23, 2026, 01:22 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 20, 2026, 03:16 PM UTC· resolved Jan 21, 2026, 12:17 AM UTC

1. investigating Jan 20, 2026, 03:16 PM UTC

   Dear VusionCloud users, We are currently investigating an issue impacting a subset of users. A large number of Subone access points are currently offline. While data integration and matchings are still being processed, label updates are currently affected in these stores. Our engineers are actively working on finding a solution. Thank you for your patience and understanding. The VusionCloud team

2. investigating Jan 20, 2026, 05:13 PM UTC

   We are still investigating the issue. Our engineers are currently narrowing down the causes for the incident, and working on a workaround. Thank you for your patience

3. identified Jan 20, 2026, 08:12 PM UTC

   The issue has been identified as a certificate issue. A workaround has been implemented and access points are coming online again.

4. monitoring Jan 20, 2026, 08:26 PM UTC

   Dear customers, A fix has been implemented and our engineers will keep monitoring. Thank you for your understanding.

5. resolved Jan 21, 2026, 12:17 AM UTC

   Dear customers, Our metrics show that the incident was fully resolved and operations resumed. No data has been lost and no action is needed from users. We thank you for your understanding during this event.

Read the full incident report →

Started Dec 04, 2025, 09:55 AM UTC· resolved Dec 04, 2025, 12:14 PM UTC

1. investigating Dec 04, 2025, 09:55 AM UTC

   Dear users, An incident is currently affecting our platform in Europe, preventing matching and integration API calls to reach VusionCloud for a subset of customers. Clients may be experiencing 504 Gateway timeout errors while trying to reach VusionCloud Our engineers are currently investigating the issue. Affected customer are encouraged to wait until the incident is resolved before trying to send the calls again.

2. identified Dec 04, 2025, 10:36 AM UTC

   The issue has been identified, and a fix has been made as a workaround. Impacted customers can now resume sending data, APIs are now fully reachable again.

3. monitoring Dec 04, 2025, 11:09 AM UTC

   The fix has been implemented and services are fully usable again. We are closely monitoring the situation.

4. resolved Dec 04, 2025, 12:14 PM UTC

   No further consequence has been detected, and all systems are fully operational again. The incident is now closed.

Read the full incident report →

Started Sep 29, 2025, 11:55 AM UTC· resolved Sep 29, 2025, 06:08 PM UTC

1. identified Sep 29, 2025, 11:55 AM UTC

   Dear customers, Our Access-Points authentication server is experiencing issues. Customers may experience difficulties configuring access-points and AP requiring tokens may not receive them in due time, causing them to be offline. We will notify you as it progresses. Thank you for your understanding.

2. identified Sep 29, 2025, 02:59 PM UTC

   Dear customers, Issue is still on-going. Our engineers are working to restore services. We will notify of any change. Thank you for your understanding.

3. monitoring Sep 29, 2025, 04:24 PM UTC

   Dear customers, A workaround has been implemented and our metrics show that Access-Points can successfully connect. We will be monitoring closely that all services are restored and will notify of any changes. Thank you for your understanding.

4. resolved Sep 29, 2025, 06:08 PM UTC

   Our metrics show that the incident has been resolved and AP can connect as expected. On behalf of VUSION Group, we would like to thank you for your understanding.

5. postmortem Nov 19, 2025, 03:50 PM UTC

   # What happened? On September 29th, 2025, a major incident occurred affecting the AP-Auth service. Access Points \(APs\) across multiple sites experienced disconnections and instability. The incident was detected due to a sudden surge in incoming network traffic, which led to saturation of the AP-Auth service and widespread connectivity issues. The support and engineering teams were alerted and began investigating the root cause. # What went wrong, and why? The incident was triggered by an unexpected and sustained spike in incoming traffic to the load balancer, with traffic volume increasing by a factor of seven compared to the previous month. This network overload saturated the AP-Auth service’s TCP connection pool, resulting in mass AP disconnections while trying to renew their jwt tokens. Further investigation revealed that the load exceeded the configured limits, which caused exhaustion of resources. There was no evidence of database or CPU bottlenecks. # How did we respond? Upon detection of the incident, the engineering team took several immediate actions: * Increased system limits for TCP thread pool size and open files. * Implemented automated restarts of the AP-Auth service at regular intervals to restore partial connectivity. * Enhanced real-time monitoring of TCP connection states and log analytics. These actions stabilized the service and eventually allowed APs to progressively reconnect. # How are we preventing this in the future? To prevent similar incidents, the following corrective actions have been implemented: * System parameters have been reviewed and increased to handle higher loads. * Real-time dashboards and alerts have been improved to monitor traffic and resource usage, enabling earlier detection of anomalies. In addition, the recent migration on November 18th transitioned the ap-auth to a new generation server that offers enhanced scalability and stability, allowing for better monitoring and higher load handling. We sincerely appreciate your patience and continued support.

Read the full incident report →

Started Jul 15, 2025, 11:55 AM UTC· resolved Jul 15, 2025, 02:07 PM UTC

1. investigating Jul 15, 2025, 11:55 AM UTC

   Dear customers, We have identified an issue with the service authenticating AP. It may cause disruptions when adding and deleting APs on VUSION Cloud. APs that disconnect and need to reconnect may stay offline as they have troubles being authenticated. We understand that such situations have major impact on your stores' operations and we are working to resolve as soon as possible. We will keep you informed as the situation progresses. Thank you for your understanding.

2. identified Jul 15, 2025, 11:57 AM UTC

   Dear customers, We have identified an issue with the service authenticating AP. It may cause disruptions when adding and deleting APs on VUSION Cloud. APs that disconnect and need to reconnect may stay offline as they have troubles being authenticated. We understand that such situations have major impact on your stores' operations and we are working to resolve as soon as possible. We will keep you informed as the situation progresses. Thank you for your understanding.

3. monitoring Jul 15, 2025, 01:23 PM UTC

   A fix has been implemented. Waiting connections are back online, and adding, updating and deleting access points is now working as expected again. We will keep monitoring the situation.

4. resolved Jul 15, 2025, 02:07 PM UTC

   The situation is now stable and services are working as expected.

Read the full incident report →

Started Jun 13, 2025, 08:46 AM UTC· resolved Jun 13, 2025, 12:42 PM UTC

1. monitoring Jun 13, 2025, 08:46 AM UTC

   Dear customers, We have identified a processing slowness issue, for items data integration as well as matchings, that is impacting a subset of customers in Europe. Our engineers have found the root cause and they have solved the issues across the two impacted instances. Our metrics show improvement after the change is applied. No data has been lost. We will continue to monitor the situation closely and keep you updated. Thank you for your understanding.

2. resolved Jun 13, 2025, 12:42 PM UTC

   Dear customers, Our metrics are showing that the incident has been resolved. On behalf of VUSION Group, the whole Cloud team would like to apologize for the incident and to thank you for your understanding. Please be assured that we are committed to continuous improvement and that we strive to provide the best experience as possible.

Read the full incident report →

Started Jun 11, 2025, 02:26 PM UTC· resolved Jun 11, 2025, 03:55 PM UTC

1. monitoring Jun 11, 2025, 02:26 PM UTC

   On June 11th, 2025, at 10:45 AM UTC, a subset of customers experienced a disruption to their Access Points (APs), which intermittently lost connection to VUSION Cloud and were offline for extended periods of time. Impacted customers had their AP targeting apa-eu-2.vusion.io. As soon as the incident was detected, our engineers identified the issue. A routing issue has been identified and a workaround was implemented. The issue was resolved at approximately 12:00 PM UTC. Please note that all operations resumed successfully after the APs went back online and that no data or transmissions were lost, and no actions are required from customers. However, a minor subset of access point are still offline and will need a manual reboot. Customers with offline access points are required to reboot them manually. Be assured that this incident is not related to the Access point update campaign and is closely monitored by our CloudOps team. We appreciate your patience and understanding as we work to resolve this issue. A post-mortem analysis will be published in the coming weeks. On behalf of SES-imagotag, the whole Cloud team would like to apologize for the incident and to thank you for your understanding. Please be assured that we are committed to continuous improvement and that we strive to provide the best experience as possible.

2. resolved Jun 11, 2025, 03:55 PM UTC

   Dear customers, Our metrics are showing that the incident has been resolved. On behalf of VUSION Group, the whole Cloud team would like to apologize for the incident and to thank you for your understanding. Please be assured that we are committed to continuous improvement and that we strive to provide the best experience as possible. A post-mortem analysis will be published in the coming weeks.

3. postmortem Jul 22, 2025, 10:59 AM UTC

   # What happened? On June 11th around 10:45am utc, a subset of customers began experiencing massive access points becoming offline. The issue was instantly identified by VusionCloud support teams and escalated to our engineering team at 12:00pm utc. Our engineers worked diligently to diagnose and address the issue. Thanks to their efforts, aps started to turn back online by 12:10 PM UTC on the same day. # What went wrong, and why? The incident was caused by an outage in the routing service on the servers targetted by url [apa-eu-2.vusion.io](http://apa-eu-2.vusion.io/). The maximum capacity for parallel connections was reached, and this created instabilities in the access points connectivity. # How did we respond? As soon as the problem was identified our Cloud engineers increased the capacity of the routing servers. Most of the devices could be recovered successfully after restarting the service, while the remainder have been recovered after a manual restart. The Cloud engineers monitored the situation closely and haven't reported further issues with Access Points then. # How are we preventing this in the future? Several actions have been taken to mitigate the impact of similar events in the future: * The size of the servers has been significantly increased. * Additionally, the service scaling capacity has been significantly increased, so the necessary number of servers are added if usage peaks. * The monitoring of Access Points connectivity to V:Cloud has been improved. Thank you for your patience and continued support.

Read the full incident report →

Started May 13, 2025, 01:53 AM UTC· resolved May 13, 2025, 06:18 AM UTC

1. identified May 13, 2025, 01:53 AM UTC

   Dear customers, We have identified a processing issue, for items data integration as well as matchings, that is impacting most customers in Europe and US environments. Our analysis has traced it to a change on Microsoft Azure. Our engineers have engaged with Azure support and are working on finding a workaround. No data has been lost. Once we recover operations, we will be able to process data. We understand that such situations have major impact in your stores' operations and we are working to resolve as soon as possible. We will keep you informed as the situation progresses. Thank you for your understanding.

2. monitoring May 13, 2025, 03:54 AM UTC

   Dear customers, Our engineers have found a workaround and they are rolling it out across instances. Our metrics show recovery after the change is applied. We will keep you informed as the workaround is deployed and data processed. Thank you for your understanding.

3. monitoring May 13, 2025, 04:26 AM UTC

   Dear customers, The change has been rolled out and our metrics are showing that the issue does not occur anymore. Additionally, our engineers are actively replaying all the matchings and item integrations that were received by the system but not executed during the disruption period. This ensures that all pending data is processed correctly without any loss. We will continue to monitor the situation closely and keep you updated. Thank you for your understanding.

4. resolved May 13, 2025, 06:18 AM UTC

   Dear customers, Metrics show full recovery and data has been processed. Transmissions are being processed as expected. We will work with Microsoft to avoid the risk of such incident and we will share the post-mortem, with necessary details. We would like to thank you for your understanding during this incident.

Read the full incident report →

Started Apr 15, 2025, 12:00 PM UTC· resolved Apr 15, 2025, 12:00 PM UTC

1. resolved Apr 16, 2025, 09:45 AM UTC

   Description: An issue has been identified in the generation of VusionLink QR codes. The QR code generation process requires access to our API management system. It was discovered that the rights of the function responsible for QR code generation on the API management system were inadvertently downgraded. This downgrade prevented the QR codes from being created successfully. Root Cause: The rights for the QR code generation function were unintentionally reduced, leading to a failure in the QR code creation process. Resolution: The necessary rights were restored on April 16, 2025, at 9:00 UTC. This action has resolved the issue, and the QR code generation process is now functioning as expected.

Read the full incident report →

Started Apr 09, 2025, 06:10 PM UTC· resolved Apr 09, 2025, 09:21 PM UTC

1. investigating Apr 09, 2025, 06:10 PM UTC

   Dear customers, An issue is currently on-going with SubOne access-points. Our metrics are showing that SubOne AP experience major disconnections. This may impact pending transmissions. Our engineers are working on this and will keep you informed. Thank you for your understanding as we work this matter.

2. monitoring Apr 09, 2025, 08:43 PM UTC

   Dear customers, Engineers have performed maintenance operations and our metrics are showing stable connections now. We will keep monitoring and will update status. Thank you for your understanding.

3. resolved Apr 09, 2025, 09:21 PM UTC

   Our metrics show that this incident has been resolved. Thank you for your understanding.

Read the full incident report →

Started Mar 18, 2025, 06:49 PM UTC· resolved Mar 18, 2025, 07:42 PM UTC

1. identified Mar 18, 2025, 06:49 PM UTC

   Dear customers, We have identified that some customers may be impacted by delays or errors in processing data, such as matchings or item updates. Our engineers have pinpointed it to Microsoft Azure network latency. We have escalated the matter to Microsoft Azure and we are monitoring closely. Our current analysis shows that no data is lost and that it will processed as soon as the issue is resolved. We will inform you of any changes. Thank you for your understanding.

2. monitoring Mar 18, 2025, 06:55 PM UTC

   Dear customers, Our metrics are showing that issue is resolved and data processing is done as expected. Our engineers will keep monitoring while we work with Microsoft to get a root cause analysis. Thank you for your understanding and continuous support, Your VUSION:Cloud team.

3. resolved Mar 18, 2025, 07:42 PM UTC

   Dear customers, our metrics are showing that the issue is resolved since one hour and all delayed data has been processed. Our engineers are working with Microsoft Azure to get a root cause on their end. We thank you for your understanding.

Read the full incident report →

Started Mar 18, 2025, 03:25 PM UTC· resolved Mar 18, 2025, 04:24 PM UTC

1. investigating Mar 18, 2025, 03:25 PM UTC

   Dear customers, We are currently facing slow integrations and matchings. Delays can be observed on Vusion Manager. We are currently investigating the issue. Thank you for your understanding.

2. identified Mar 18, 2025, 03:59 PM UTC

   The issue has been identified as a probable issue from this morning's release. The version has been rolled back to the previous one.

3. monitoring Mar 18, 2025, 04:07 PM UTC

   rollback seems effective. Monitoring is still ongoing.

4. monitoring Mar 18, 2025, 04:13 PM UTC

   We are continuing to monitor for any further issues.

5. resolved Mar 18, 2025, 04:24 PM UTC

   The rollback fixed the issue. Situation is back to normal. We will investigate the root cause and provide a fix. Thank you for your understanding.

6. postmortem Mar 20, 2025, 03:26 PM UTC

   ## What happened ? On March 18th at 9:00 AM UTC a new release of the VusionCloud platform was deployed in the US environment. The update was completed within the scheduled maintenance window, and initial routine checks - including matching processes and item integration - were successful. VusionCloud engineers closely monitored the system until the maintenance window closed at 11:00 AM UTC. However, a few hours later, delays began accumulating in the matching processes and item data ingestion. The issue was escalated to VusionCloud engineers at around 3:00 PM UTC. The team promptly investigated and identified the new release as the root cause. To restore the normal operations, the faulty version was swiftly rolled back, and operations were back to normal by approximately 3:30 PM UTC. ## What went wrong, and why? The March release of VusionCloud included an update to the library responsible for reading and processing messages from queueing services, as the library currently used in production is set to be deprecated soon. To ensure that all our production libraries remain up to date, well-maintained, and capable of delivering the best security and performance, we made the decision to upgrade to the latest version. However, this update introduced changes to the library’s configuration parameters, which were not correctly implemented. The new version altered the library’s standard behavior, causing improper application scaling under increased load starting at 12:30 PM UTC. As a result, messages began to accumulate leading to processing delays. Once our engineers identified the issue, they promptly rolled-it back and found out that the update was the root cause. System performance began improving immediately by 3:30 PM UTC, and all services were fully restored by 4:30 PM UTC. ## How did we respond? First delays appeared at 12:30 PM UTC when the application failed to scale effectively under the rising load. Because the delays accumulated gradually and did not trigger exceptions or errors, our proactive alerts remained below the threshold for detection. While messages were processed successfully, processing time continued to increase. Our engineering team became aware of the issue around 3:00PM UTC, at the same time it was escalated internally by our support team. Given the impact on performance, they decided to allocate a fixed time to find a potential fix before proceeding with a rollback. As no immediate solution was found, they initiated the rollback at 3:30 PM UTC to minimize disruption to the production environment. Performance began improving immediately after the rollback, with 80% of the delays resolved within 20 minutes. By 4:30 PM UTC, all services had returned to normal. Once services stabilized, the team conducted a thorough investigation and quickly identified the root cause of this incident, the update of the new library. ## How are we making incidents like this less likely or less impactful? We remain committed to reducing the likelihood of such incidents and minimizing their impact. * To strengthen our monitoring, we will reinforce post-maintenance checks and closely track system behavior during the first peak activity periods of the day. * Additionally, we will significantly lower alert thresholds to ensure they are triggered at the first signs of delays, allowing for faster detection and response. * To prevent similar occurrences, we will enhance our automated testing by incorporating these specific production conditions. This will ensure that each release is tested under those operational scenarios before deployment. * Finally, a fix will be deployed to ensure seamless compatibility with the updated library’s standard behavior. Thank you for your continued support and understanding.

Read the full incident report →

Started Feb 05, 2025, 09:42 AM UTC· resolved Feb 05, 2025, 04:04 PM UTC

1. identified Feb 05, 2025, 09:42 AM UTC

   Dear customers, We have identified delays in displaying information on Vusion Manager, and issues in labels status synchronization on Vusion Manager UI. The cause has been identified, and our engineers are actively working on fixing the situation. Please note that no data has been lost, and operations in the stores, such as updates and flash, are not impacted. On behalf of VusionGroup, the whole Cloud team would like to apologize for the incident and to thank you for your understanding

2. identified Feb 05, 2025, 10:16 AM UTC

   Delays and latency on Manager UI are now fixed. Our engineers are still rolling out the fix to prevent the same issue to occur in the future.

3. monitoring Feb 05, 2025, 11:03 AM UTC

   The fix has been rolled out. The situation is back to normal. We will be closely monitoring the situation for the next hours. Thank you for your understanding.

4. resolved Feb 05, 2025, 04:04 PM UTC

   The incident has been resolved. All services are back to normal, including the items research.

Read the full incident report →

Started Jan 09, 2025, 10:01 AM UTC· resolved Jan 09, 2025, 01:14 PM UTC

1. investigating Jan 09, 2025, 10:01 AM UTC

   Dear customers, A subset of customers using the endpoint ap-eu.vusion.io:64173 are currently experiencing strong delays in image transmissions. Our engineers are actively working on identifying the issue and restore services. Please note that no data has been lost, all transmissions are queued and will be processed eventually. On behalf of VusionGroup, the whole Cloud team would like to apologize for the incident and to thank you for your understanding. Please be assured that we are committed to continuous improvement and that we strive to provide the best experience as possible.

2. identified Jan 09, 2025, 10:21 AM UTC

   Dear customers, The cause for the delay has been identified as a high resource consumption. The server has been restarted and waiting transmissions are being processed. Our teams are closely monitoring the situation.

3. monitoring Jan 09, 2025, 10:41 AM UTC

   Waiting transmissions are still catching up, the majority has been processed already. Our teams are still closely monitoring.

4. resolved Jan 09, 2025, 01:14 PM UTC

   Dear customers, The issue on endpoint ap-eu.vusion.io:64173 has been resolved. All the delays have been resorbed, and all waiting tasks have finished. The endpoint is now healthy again and services resumed to normal. On behalf of VusionGroup, the whole Cloud team would like to apologize for the incident and to thank you for your understanding.

5. postmortem Jan 27, 2025, 05:36 PM UTC

   ## What happened ? Starting on January 9th starting from 4:15 AM UTC, multiple clients experienced access points going offline or delays in transmissions on endpoint [ap-eu.vusion.io:64173](http://ap-eu.vusion.io:64203/). This outage caused business disruptions and temporarily halted transmissions in stores. Once the issue was identified, the VusionCloud team swiftly took corrective action to restore service and planned for a long term fix. ## What went wrong, and why? On January 9th, the endpoint [ap-eu.vusion.io:64173](http://ap-eu.vusion.io:64203/) started consuming an abnormally high amount of resources. As a result, endpoint performances were degraded, eventually causing access points to disconnect due to resource exhaustion. VusionCloud engineers' analysis showed that the endpoint has been populated with a large number of stores. From 4:15 AM UTC on January 9th, a high activity on many of these stores made the server reach its capacities, which eventually led to the degradation of service. ## How did we respond? As soon as the first alerts were raised, VusionCloud engineers started to investigate the issue and took corrective action to improve the situation. All manual attempts only got limited and temporary results. The server was finally restarted at 8:40 AM UTC, leading the ressources to be cleared and reallocated. The access points came back online and full service was restored from 9:10 am. ## How are we making incidents like this less likely or less impactful? We are committed to minimizing the likelihood of such events and mitigating their impact. As a short term fix, the server was restarted and extra monitoring has been done on this endpoint. As a mid term fix, the server capacity will be increased during January release to make sure a high load does not cause a similar problem in the future. As a long-term fix, the endpoint [ap-eu.vusion.io:64173](http://ap-eu.vusion.io:64203/) has been flagged as prioritary for migrations to our new endpoints, which are more resilients to such loads and that will allow for better load balancing. We sincerely apologize for any inconvenience this service disruption may have caused. We understand the impact on your operations and remain dedicated to continuously improving our services. On behalf of VusionGroup, we thank you for your continued support and understanding.

Read the full incident report →

Started Nov 18, 2024, 11:07 AM UTC· resolved Nov 18, 2024, 11:56 AM UTC

1. investigating Nov 18, 2024, 11:07 AM UTC

   We are aware of an issue causing Vision Link to display offline messages. Our team is actively investigating and working to resolve the problem as quickly as possible. We apologize for any inconvenience this may cause and appreciate your patience. Further updates will be provided as soon as they are available. Thank you for your understanding.

2. monitoring Nov 18, 2024, 11:19 AM UTC

   A fix has been implemented for the issue causing Vision Link to display offline messages and for matchings that were not functioning correctly. Our team is actively monitoring the system to ensure stability and optimal performance. We appreciate your patience as we worked to address these problems. If you experience any further issues, please don’t hesitate to contact us. Thank you for your understanding.

3. resolved Nov 18, 2024, 11:56 AM UTC

   Dear customers, Our metrics are showing that the workaround implemented has proven effective. Our engineers will keep monitoring the situation and investigating as to the root cause. We have identified an issue on Azure back-ends, and we are escalating to understand the cause. A postmortem report will be provided later. On behalf of VUSION group, please accept our apologies. We would like to thank you for your understanding.

Read the full incident report →

Started Nov 16, 2024, 06:42 AM UTC· resolved Nov 16, 2024, 08:44 PM UTC

1. investigating Nov 16, 2024, 06:42 AM UTC

   Dear customers, Since midnight, 16th of November, Meraki access-points are progressively disconnecting from VUSION Cloud. Our engineers are investigating the case. We will keep you informed as the situation evolves. On behalf of VUSION group, we thank you for your understanding.

2. identified Nov 16, 2024, 08:46 AM UTC

   Dear customers, We have engaged with Meraki support and engineers are working on it. We will keep you informed as it progresses. Thank you for your understanding.

3. identified Nov 16, 2024, 10:24 AM UTC

   Dear Customers, We want to provide you with the latest update regarding the ongoing issue. Meraki engineers have identified a problem on their backend and are actively working to resolve it. Engineers from both our team and the Meraki team are collaborating closely, and this has been classified as a Priority 1 (P1) for both organizations. We will share additional updates as soon as more information becomes available. Thank you for your patience and understanding as we work to address this matter.

4. identified Nov 16, 2024, 11:14 AM UTC

   Dear Customers, We’re pleased to inform you that Meraki engineers have implemented a fix for the identified issue, and it is currently being deployed across all affected systems. The server update on Meraki’s side may take up to two hours to complete, and it may take a few additional hours for all access points (APs) to fully recover. However, no action is required on your side. We’d also like to reassure you that no data updates have been lost during this period. Data transmission will automatically resume as the tags come back online. Our teams are actively monitoring the situation to ensure a smooth restoration. We will keep you informed of any further developments. Thank you for your understanding.

5. monitoring Nov 16, 2024, 11:43 AM UTC

   Dear Customers, We’re pleased to inform you that Meraki engineers have implemented a fix for the identified issue, and it is currently being deployed across all affected systems. The server update on Meraki’s side may take up to two hours to complete, and it may take a few additional hours for all access points (APs) to fully recover. However, no action is required on your side. We’re happy to report that we are already seeing many access points (APs) coming back online. As they do, data transmissions are resuming as expected. We’d also like to reassure you that no data updates have been lost during this period. Our teams are actively monitoring the situation to ensure a smooth restoration and will keep you informed of any further developments. Thank you for your understanding.

6. monitoring Nov 16, 2024, 12:44 PM UTC

   Dear Customers, We’re pleased to inform you that the majority of access points (APs) are now back online, and data transmissions have resumed as expected. Our teams are actively monitoring the situation to ensure a smooth and complete restoration. Thank you for your understanding.

7. resolved Nov 16, 2024, 08:44 PM UTC

   Dear Customers, We’re pleased to inform you that 99% of access points (APs) are now back online, and data transmissions have resumed as expected. For the remaining APs, Meraki is actively working on their recovery. If you are still experiencing issues, we encourage you to open a ticket directly with Meraki support for further assistance. As the situation is nearly resolved, we will be closing this incident. Thank you for your patience and understanding throughout this process.

Read the full incident report →