Singlewire Software Outage History

Started Oct 15, 2026, 02:59 PM UTC· resolved Oct 15, 2026, 03:23 PM UTC

1. investigating Oct 15, 2026, 02:59 PM UTC

   We have received reports from RapidSOS of elevated errors that may prevent Emergency Calling from working with InformaCast. For additional information, please visit: https://status.rapidsos.com/ We will update this incident when more information is made available.

2. resolved Oct 15, 2026, 03:23 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 05, 2026, 10:26 PM UTC· resolved Oct 07, 2026, 01:43 PM UTC

1. identified Oct 05, 2026, 10:26 PM UTC

   iOS users who start a Mobile Panic Button or choose a response that turns on location tracking are not having their location shared, and responders will not see them on the tracking map. A root cause has been identified and we plan to deploy a fix as soon as possible. We apologize for the disruption in service.

2. monitoring Oct 06, 2026, 06:37 PM UTC

   A fix has been implemented and we are monitoring the results.

3. monitoring Oct 06, 2026, 06:37 PM UTC

   We are continuing to monitor for any further issues.

4. resolved Oct 07, 2026, 01:43 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 01, 2026, 05:11 PM UTC· resolved Sep 01, 2026, 06:34 PM UTC

1. identified Sep 01, 2026, 05:11 PM UTC

   We have received reports from Twilio of elevated errors that may prevent InformaCast from sending SMS and phone calls. Consider sending through other channels (such as email) for critical notifications. For additional information, please visit: https://status.twilio.com/ We will update this incident when more information is made available.

2. monitoring Sep 01, 2026, 06:12 PM UTC

   A fix has been implemented and telephony services are operating normally. We will continue to monitor for any further impact.

3. resolved Sep 01, 2026, 06:34 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 13, 2026, 08:47 PM UTC· resolved Aug 14, 2026, 09:41 PM UTC

1. investigating Aug 13, 2026, 08:47 PM UTC

   We've detected slowdowns and potential failures in notification delivery across many channels. Those channels include: - SMS messages - Phone calls - Emails - Push notifications to mobile apps - All on-premises devices, including speakers This may also impact authentication and general platform access. Our engineers are working to understand the scope of the issue and find a fix. We'll provide an update when we know more.

2. monitoring Aug 13, 2026, 10:07 PM UTC

   We've failed over on the impacted service, and we believe service should be restored at this point. We'll continue to monitor overnight and provide an update in the morning when we have more information about the root cause of the service disruption.

3. monitoring Aug 14, 2026, 01:54 PM UTC

   We're continuing to monitor to ensure we've completely recovered, and we'll give another update before 2PM CDT.

4. monitoring Aug 14, 2026, 06:40 PM UTC

   Service has been restored, and we believe we've addressed the root cause. However, because of the intermittent nature of the issue, we're going to continue to monitor for the rest of the business day.

5. resolved Aug 14, 2026, 09:41 PM UTC

   This incident has been resolved.

6. postmortem Aug 24, 2026, 05:07 PM UTC

   On August 13th, we observed a few isolated occurrences of queries to our in-memory data store timing out. While we don't have a definitive root cause for the issue, we have been able to simulate similar cases with high query volume in our testing environments, and are making the following remediations to both hopefully prevent this issue from occurring again as well as mitigate it in the event that it does. First, we'll be upgrading the instance class of the machines this database runs on, which uses a more modern AWS hypervisor and processor architecture. Additionally, we'll be bolstering certain commands with retry and circuit breaker logic, such that in the rare case that this does happen again, the command will be immediately retried to a point, while allowing the queries to be skipped entirely if the database is truly unavailable. Lastly, we'll be tweaking our slow query logs in order to ensure we keep an eye on performance of this resource so that we can continue to tune our queries and ensure they don't place an undue amount of stress on this database. We apologize for any inconvenience this may have caused.

Read the full incident report →

Started Jul 30, 2026, 03:41 PM UTC· resolved Jul 30, 2026, 09:42 PM UTC

1. investigating Jul 30, 2026, 03:41 PM UTC

   We are currently investigating this issue.

2. identified Jul 30, 2026, 05:19 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Jul 30, 2026, 08:03 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Jul 30, 2026, 09:42 PM UTC

   This incident has been resolved.

5. postmortem Jul 31, 2026, 01:44 PM UTC

   **What Happened** On July 30, 2026, non-administrative users who should have access to edit distribution lists could not do that via the InformaCast web application. Access to update distribution lists through the API or by administrative users was not impacted. **Root Cause** We made changes to the distribution lists pages that were released on July 29, 2026. These changes were intended as a simple update to the form, but they introduced a request to retrieve information inaccessible to most non-administrator users, including those with the Notification Sender permissions set. **Resolution** We updated the form to correctly gate information intended for administrators, which in turn restored the ability for non-administrators to access the form. To safeguard against similar issues with our permissions in the future, we will be updating our automated testing to better cover non-administrator access throughout our application.

Read the full incident report →

Started Jun 29, 2026, 04:34 PM UTC· resolved Jun 29, 2026, 06:52 PM UTC

1. investigating Jun 29, 2026, 04:34 PM UTC

   We are currently experiencing an outage on app.visitor-aware.com. Attempts to access Visitor Aware result in the messages "Internal Service Error" or "Service Unavailable." We are investigating the root cause and will update this incident as we learn more.

2. monitoring Jun 29, 2026, 05:38 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jun 29, 2026, 06:52 PM UTC

   This incident has been resolved.

4. postmortem Jul 01, 2026, 05:10 PM UTC

   **What happened** On June 29, 2026, users attempting to log in to the Visitor Access web portal experienced performance degradation and timeouts. The Check In Kiosk iOS app was not affected. **Root cause** The login permission cache was storing too much data in a single entry, which exceeded the maximum allowed request size and made the web portal effectively unavailable. **Resolution** We have increased the granularity of our caching, which resolves the issue and improves overall performance for permission-gated requests throughout the application.

Read the full incident report →

Started May 29, 2026, 07:25 AM UTC· resolved May 29, 2026, 12:32 PM UTC

1. investigating May 29, 2026, 07:25 AM UTC

   We have received reports from RapidSOS of elevated errors that may prevent Emergency Calling from working with InformaCast. For additional information, please visit: https://status.rapidsos.com/ We will update this incident when more information is made available.

2. resolved May 29, 2026, 12:32 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 26, 2026, 05:11 PM UTC· resolved May 26, 2026, 06:34 PM UTC

1. investigating May 26, 2026, 05:11 PM UTC

   We have received reports from RapidSOS of elevated errors that may prevent Emergency Calling from working with InformaCast. For additional information, please visit: https://status.rapidsos.com/ We will update this incident when more information is made available.

2. investigating May 26, 2026, 06:34 PM UTC

   We are continuing to investigate this issue.

3. resolved May 26, 2026, 06:34 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 08, 2026, 10:10 AM UTC· resolved May 08, 2026, 01:34 PM UTC

1. investigating May 08, 2026, 10:10 AM UTC

   We're investigating an intermittent issue with SMS message delivery

2. resolved May 08, 2026, 01:34 PM UTC

   This incident has been resolved.

3. postmortem May 08, 2026, 06:12 PM UTC

   Our monitoring system triggered an alert that indicated a potential service disruption to SMS delivery. After investigation, we confirmed that all systems were operating normally throughout this period. The alert was caused by overly aggressive monitor thresholds, not an actual service issue. We are tuning our alerting to reduce false positives.

Read the full incident report →

Started May 07, 2026, 01:41 AM UTC· resolved May 07, 2026, 04:13 AM UTC

1. identified May 07, 2026, 01:41 AM UTC

   We have received reports from RapidSOS of elevated errors that may prevent Emergency Calling from working with InformaCast. For additional information, please visit: https://status.rapidsos.com/ We will update this incident when more information is made available.

2. investigating May 07, 2026, 01:45 AM UTC

   We are currently investigating this issue.

3. monitoring May 07, 2026, 01:50 AM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved May 07, 2026, 04:13 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 09, 2026, 03:41 PM UTC· resolved Mar 10, 2026, 02:07 PM UTC

1. monitoring Mar 09, 2026, 03:41 PM UTC

   We're experiencing an issue where events from on-premises servers may be delayed from reaching the cloud. This may include events intended to broadcast notifications to the cloud, such as using a phone to activate a DialCast Dialing Configuration. This is also presently impacting notification progress and completion reporting, as well as display of IDNs. For now, to ensure notification delivery to cloud devices, activating notifications from the web application or other cloud devices is encouraged. We've implemented a fix, and we're monitoring the results.

2. monitoring Mar 09, 2026, 06:39 PM UTC

   We are continuing to monitor. At this point, notifications sent from on-premises servers should be received by the cloud and distributed correctly. Events related to on-premises devices such as speakers and IDNs are still being processed; there may be some delay before changes to such devices are visible in the cloud.

3. monitoring Mar 09, 2026, 08:24 PM UTC

   Device updates are still facing delays, but we are starting to see our event processor catch up. We'll continue to monitor this overnight and update this tomorrow, 3/10.

4. resolved Mar 10, 2026, 02:07 PM UTC

   This incident has been resolved.

5. postmortem Mar 10, 2026, 05:52 PM UTC

   On Friday night, we deployed a small infrastructure change that required a reboot of our database. During this reboot, we encountered a bug in our event processor code that caused events not to be processed until service reboot. To compound this, our monitoring platform had invalidated the alarm we had in place to monitor errors with our event processor. This caused us to remain unaware of the issue until Monday morning, where we promptly fixed it by rebooting the service containing the event processors. At that point, the platform needed to catch up by processing all the events that had been building up over the weekend, which completed on Monday night. We will be taking a multi-pronged approach to ensuring this doesn’t happen again: 1. We have fixed our event monitors, and will be testing them every release going forward to ensure they remain operational. We are also following up with our monitoring platform to ensure future monitoring will not be disrupted. 2. We are augmenting our event processing code in the following ways: 1. We have switched to a more robust event processing pattern that is not vulnerable to the bug that caused consumers to stop polling after a database reboot. 2. We have increased the processing throughput such that if this _were_ to happen again, our consumers would catch up much more quickly than they did during this incident. We apologize for service disruptions caused by this incident.

Read the full incident report →

Started Nov 18, 2025, 01:39 PM UTC· resolved Nov 18, 2025, 08:20 PM UTC

1. investigating Nov 18, 2025, 01:39 PM UTC

   Visitor Aware is currently down due to issues with our third party provider Cloudflare. They are working to get the issued resolved and we will keep you updated on their progress.

2. identified Nov 18, 2025, 01:44 PM UTC

   The issue has been identified and a fix is being implemented.

3. identified Nov 18, 2025, 02:45 PM UTC

   Cloudflare has implemented a fix. We will continue to post updates as we monitor their progress.

4. monitoring Nov 18, 2025, 03:47 PM UTC

   Cloudflare appears to resolved their issue but not yet given the all clear. We will continue to monitor their progress.

5. monitoring Nov 18, 2025, 04:45 PM UTC

   We are continuing to monitor Cloudflare as they make progress on their issue.

6. monitoring Nov 18, 2025, 05:43 PM UTC

   Cloudflare is continuing to make progress on their recovery.

7. monitoring Nov 18, 2025, 07:13 PM UTC

   We are still waiting to hear back from Cloudflare that they have resolved all issues. All indications are that everything is back to normal.

8. resolved Nov 18, 2025, 08:20 PM UTC

   The Cloudflare issue has been resolved.

Read the full incident report →

Started Oct 20, 2025, 07:28 PM UTC· resolved Oct 21, 2025, 12:16 AM UTC

1. investigating Oct 20, 2025, 07:28 PM UTC

   Visitor Aware is currently experiencing extended processing times and inconsistent facial recognition matching due to the ongoing cloud provider outage. We are monitoring the situation and will update when we learn more.

2. monitoring Oct 20, 2025, 07:31 PM UTC

   We are monitoring the situation and will update when we learn more.

3. monitoring Oct 20, 2025, 11:22 PM UTC

   Our cloud provider is reporting that their service has recovered. While there may be some increased latency, we believe service should be back to normal soon. We'll continue to monitor for the next hour.

4. resolved Oct 21, 2025, 12:16 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 20, 2025, 04:30 PM UTC· resolved Oct 20, 2025, 04:30 PM UTC

1. resolved Oct 20, 2025, 08:59 PM UTC

   An increase of errors was detected starting shortly after 11:30am CDT. These failures were linked to the ongoing cloud provider outage.

Read the full incident report →

Started Oct 20, 2025, 08:53 AM UTC· resolved Oct 20, 2025, 11:03 AM UTC

1. monitoring Oct 20, 2025, 08:53 AM UTC

   InformaCast Fusion is currently experiencing a service interruption due to a widespread incident reported by our cloud provider. At this time Fusion cannot send SMS messages, place phone calls, or send email. Our engineers are aware of the issue and are continuing to monitor the situation closely. We will update this incident when we learn more.

2. monitoring Oct 20, 2025, 09:55 AM UTC

   InformaCast Fusion is now able to send SMS messages and place calls reliably. Email send are succeeding, but inconsistent delivery times are noted. We will update this incident when we learn more.

3. monitoring Oct 20, 2025, 10:05 AM UTC

   All Fusion notification channels are responding normally. We will continue to monitor the situation for next hour to ensure there are no lingering issues.

4. resolved Oct 20, 2025, 11:03 AM UTC

   All InformaCast Fusion functionality has been restored and returned to normal.

Read the full incident report →

Started Sep 17, 2025, 02:33 PM UTC· resolved Sep 17, 2025, 04:57 PM UTC

1. investigating Sep 17, 2025, 02:33 PM UTC

   Visitor Aware is currently experiencing an issue with badges not being automatically printed through the Desktop Print Helper app. This impacts version 2.1.3 of Print Helper and 12.2.6 of the web portal. The team is investigating the issue.

2. investigating Sep 17, 2025, 04:56 PM UTC

   With the release of version 12.2.6 last night, an update to the format of our visitor and student badges caused an error with current and previous Desktop Print Helper Versions (2.1.3 and previous). This resulted in certain dashboard updates to fail, and caused badges to not print automatically when using Desktop Print Helper to process print jobs. This change has been reverted so that Desktop Print Helper continues to work with version 12.2.6. To address this, we will update the badge formatting fix so that it is compatible with previous Desktop Print Helper versions.

3. resolved Sep 17, 2025, 04:57 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 02, 2025, 11:35 PM UTC· resolved Sep 03, 2025, 02:23 PM UTC

1. investigating Sep 02, 2025, 11:35 PM UTC

   We have identified an issue where some android devices are not receiving push notifications as a recipient of notifications in InformaCast Fusion. The issue is currently still under investigation.

2. monitoring Sep 02, 2025, 11:49 PM UTC

   We have swapped out a copy of one of our services, as we believe the issue is transient at this time. Both monitoring and root cause analysis are ongoing. We will resolve the incident when we are confident that Android pushes are no longer failing.

3. resolved Sep 03, 2025, 02:23 PM UTC

   This incident has been resolved.

4. postmortem Sep 04, 2025, 03:41 PM UTC

   During our investigation of this issue, we discovered a programming error with our message delivery code, that, in certain instances, caused it to stall indefinitely. We have isolated the issue and are deploying a fix to prevent it from happening again. We apologize for service disruptions caused by this incident.

Read the full incident report →

Started Aug 21, 2025, 09:45 PM UTC· resolved Sep 09, 2025, 02:57 PM UTC

1. monitoring Aug 21, 2025, 09:45 PM UTC

   Users attempting to use the Microsoft Teams integration with InformaCast to send scenarios may experience a problem where dropdowns display options from previous questions in the sending workflow rather than the current question. This prevents the user from selecting a valid response and sending the scenario. We've investigated and currently believe this to be an issue within recent versions of Microsoft Teams clients, not an error within our integration. We'll continue to monitor and investigate, and we'll update here when we have more information.

2. monitoring Sep 03, 2025, 02:50 PM UTC

   Our engineers are in correspondence with Microsoft support about this issue. At this time, scenario sending from Microsoft Teams clients is still impacted. We'll update when we know more.

3. resolved Sep 09, 2025, 02:57 PM UTC

   Microsoft has confirmed the unexpected behavior while sending scenarios from their Teams clients is a defect within their system. Because the issue is external to us, we don't have a timeline for this issue to be resolved. Please refer to this Singlewire Knowledge Base article for the most up-to-date status on this issue: https://support.singlewire.com/s/article/Unable-to-send-scenario-from-Microsoft-Teams-because-scenario-question-displays-incorrect-options

Read the full incident report →

Started Jul 31, 2025, 01:52 PM UTC· resolved Aug 01, 2025, 11:55 PM UTC

1. investigating Jul 31, 2025, 01:52 PM UTC

   We're experiencing an issue where users may be updated by the local user loader unintentionally, which may cause them to be removed from security groups and distribution lists. Our development team is working to identify the root cause and implement a solution. Any customer using the local user loader may be affected by this issue, so we recommend pausing any local user loader automation and holding off on manual updates to load requests. Cloud user loaders (Entra, School Information Systems) are not impacted by this issue. We'll update this incident when we have identified a root cause or within three hours.

2. identified Jul 31, 2025, 02:03 PM UTC

   We have identified the problem and are implementing a fix. To minimize additional impact until the fix is complete, we are temporarily disabling all local user loaders.

3. identified Jul 31, 2025, 05:02 PM UTC

   We have identified the root cause of the issue and are testing a fix for it. All local user loaders remain non-functional as a precaution until we have finished the fix. We are working to identify and restore user data that was erroneously changed or deleted.

4. monitoring Jul 31, 2025, 08:52 PM UTC

   We have deployed a change that fixes the issue and re-enabled local user loaders. We continue to work towards restoring user data that was erroneously changed or deleted.

5. resolved Aug 01, 2025, 11:55 PM UTC

   Although the issue with the Local User Loader has been corrected, our Support team will be reaching out to affected customers to help restore their data in the next few days.

Read the full incident report →

Started Jun 03, 2025, 06:42 PM UTC· resolved Jun 03, 2025, 09:51 PM UTC

1. investigating Jun 03, 2025, 06:42 PM UTC

   We are investigating reports of an issue involving WebEx Calling that is preventing new devices from registering. Modification of existing devices will prevent them from initiating scenarios.

2. investigating Jun 03, 2025, 06:46 PM UTC

   We are continuing to investigate this issue.

3. monitoring Jun 03, 2025, 07:49 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Jun 03, 2025, 09:51 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 30, 2025, 02:49 PM UTC· resolved Jun 02, 2025, 02:10 PM UTC

1. investigating May 30, 2025, 02:49 PM UTC

   Visitor Aware is currently experiencing an issue with emails not being sent. This includes automatic emails sent from the system. The team is investigating the issue.

2. investigating May 30, 2025, 07:17 PM UTC

   We are continuing to investigate issues with Visitor Aware email sends. Our next update to this incident will happen at 4pm central time or when we have identified a solution.

3. investigating May 30, 2025, 09:05 PM UTC

   We are continuing to investigate issues with Visitor Aware email sends. Our next update to this incident will happen at 6pm central time or when we have identified a solution.

4. resolved Jun 02, 2025, 02:10 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 11, 2025, 04:59 PM UTC· resolved Apr 11, 2025, 06:15 PM UTC

1. investigating Apr 11, 2025, 04:59 PM UTC

   We are currently experiencing degraded performance on VisitorAware.com. We are investigating root cause and will update this incident as we learn more.

2. resolved Apr 11, 2025, 06:15 PM UTC

   This incident has been resolved.

3. postmortem Apr 17, 2025, 07:45 PM UTC

   We disabled all SFTP file processing to address performance issues, which caused a backlog of files that attempted to process simultaneously during peak usage. This overloaded the database and resulted in timeout errors. To address this, we gradually re-enabled SFTP processing to run during off hours and increased the database size for higher memory and performance.

Read the full incident report →

Started Feb 12, 2025, 11:36 PM UTC· resolved Feb 13, 2025, 02:55 PM UTC

1. identified Feb 12, 2025, 11:36 PM UTC

   Notifications sent using a message template or scenario in a non-global site context and initiated via an on-premises trigger such as DialCast, CallAware, Night Bell, or M2M are currently not delivering to cloud or mobile recipients, nor are they delivering to any on-premises recipients that would be activated by a different Fusion server than the initiating server. On-premises recipients will still receive notifications if the activating Fusion server is the same as the initiating server or the notification is using a message template or scenario from a global site context.

2. monitoring Feb 13, 2025, 01:08 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Feb 13, 2025, 02:55 PM UTC

   This incident has been resolved.

4. postmortem Feb 14, 2025, 08:23 PM UTC

   When testing some permissions changes to our site-based permissions model, we failed to account for situations where initiation methods that come from sources other than users, such as an on-premise device, interact with message templates or scenarios assigned to a particular site. In these cases, we should take a permissive approach to initiating the notification rather than disallowing the interaction. We rolled this change back in the short-term to ensure that this workflow would continue to work. ‌ We are taking steps to ensure that this use case is covered in our automation as we attempt to reapply this change.

Read the full incident report →

Started Feb 12, 2025, 04:17 PM UTC· resolved Feb 12, 2025, 04:24 PM UTC

1. resolved Feb 12, 2025, 06:13 PM UTC

   Some bell schedules fired on 2/12/2025 between 10:17 AM CST and 10:24 AM CST may not be present for review in notification history in the InformaCast web console. This did not affect bell schedule activation for any on-premise devices.

Read the full incident report →

Started Jan 15, 2025, 04:20 PM UTC· resolved Jan 15, 2025, 04:20 PM UTC

1. resolved Jan 22, 2025, 09:15 PM UTC

   Notifications initiated from the web, mobile apps, or any cloud-side inbound integration may have failed to deliver to on-prem recipients between approximately 10:21 AM and 10:33AM on 1/15/2025.

2. postmortem Jan 22, 2025, 09:15 PM UTC

   During this window, one of our engineers was executing a scheduled InformaCast deployment. During this deployment, we encountered an unexpected error. The established procedure during such an event is to immediately escalate to a senior engineer to assess the error and its potential impact. This process was not followed and remediation steps were taken instead, which temporarily left InformaCast with too few copies of a critical service for cloud to on-premises notification to function. Steps are being taken to reinforce training around deployment communications, escalating to senior engineers or management when the unexpected occurs, and the importance of following established procedures.

Read the full incident report →