Networking Hardware Issues Discovered

Update timeline

1. investigating Dec 10, 2024, 08:37 PM UTC

   Addition issues were discovered with one of the edge firewalls in our North America East location. This a continuation of the issues discovered yesterday, December 9th, and we will be working to restore full functionality. At this time, all systems are up and operational, however there may be brief moments of dropped packets in flight between this cluster and the internet. We hope to have an update within the hour.

2. identified Dec 10, 2024, 08:59 PM UTC

   Jenzabar Cloud Infrastructure has identified and isolated improperly functioning hardware and is currently working with IBM support to take corrective action. Additionally, an unexpected failover event took place at the following time: 2024/12/10 15:11:37 EST Cloud infrastructure isolated the bad hardware and performed another failover at: 2024/12/10 15:47:12 Disconnects to hosted resources could have occurred during the times mentioned above. At this time, we expect all services to remain stable. Please stay tuned as we will continue to provide updates as they are available.

3. monitoring Dec 11, 2024, 02:47 AM UTC

   Work is still being performed in the IBM data center. All systems have remained stable since the last update and are expected to remain stable at this time. We will continue to monitor the status of this and will provide another update when it is available.

4. monitoring Dec 11, 2024, 01:41 PM UTC

   Update: As of 2024/12/11 04:30 EST, corrective changes were completed by IBM technicians. Since that time, systems have been monitored throughout this event and no further failover conditions have taken place since the last one yesterday, at 2024/12/10 15:47:12. Jenzabar Cloud Infrastructure will continue to monitor the environment for the next 24 hours before marking this incident as resolved. If you have any questions, concerns, or issues to report, please report them through normal channels at this time. Jenzabar Cloud Infrastructure

5. resolved Dec 12, 2024, 01:42 PM UTC

   At this time, all hardware is in great standing in our North America East data center location and no further corrective action is needed within the cluster. Marking this incident as resolved.