LUMASERV incident

Network outage

Resolved after 1 hour 15 minutes, affecting Core Network, on February 16, 2026.

postmortem Feb 17, 2026, 05:48 PM UTC

On February 16, 2026, an incident identified as INC-262 occurred, resulting in a critical network outage that affected all systems in our Frankfurt data centers for a duration of 8 minutes. The incident was triggered by a routine configuration change made to a static route on one of our routers. Prior to implementation, the configuration underwent validation via the operating system \(Juniper Junos OS\), which generally checks for errors. However, due to a bug in the firmware, the system incorrectly accepted a faulty configuration change. 4 minutes after the change was applied, both forwarding engines were reported unreachable by the routing engine. The issue was promptly detected and reported by our monitoring systems. An automatic rollback occurred one minute following the failure due to a lack of manual confirmation, which restored both routing engines and forwarding engines. Safety mechanisms have worked as expected and resulted in a quick recovery completed by 7:27 PM. To preemptively harden our infrastructure against similar issues in the future, we already planned to transition from a redundant routing chassis to completely isolated devices, ensuring full redundancy even in case of such failures. This transition is scheduled for April/May during a planned maintenance. We appreciate your understanding and patience as we work to enhance our systems' reliability.

Started
Feb 16, 2026, 06:15 PM UTC
Resolved
Feb 16, 2026, 07:30 PM UTC
Duration
1h 15m
Detected by Pingoru
Feb 16, 2026, 06:15 PM UTC

Affected components

Core Network

Update timeline

  1. investigating Feb 16, 2026, 06:19 PM UTC

    We are currently experiencing a network outage at our location in Frankfurt. We are currently investigating and provide updates on this page as soon as possible.

  2. identified Feb 16, 2026, 06:30 PM UTC

    The network is available again since 19:24 CET. We continue investigating and analyzing the route cause.

  3. resolved Feb 17, 2026, 09:15 AM UTC

    The incident has been resolved and no further impact is expected. We will provide details today afternoon.

  4. postmortem Feb 17, 2026, 05:48 PM UTC

    On February 16, 2026, an incident identified as INC-262 occurred, resulting in a critical network outage that affected all systems in our Frankfurt data centers for a duration of 8 minutes. The incident was triggered by a routine configuration change made to a static route on one of our routers. Prior to implementation, the configuration underwent validation via the operating system \(Juniper Junos OS\), which generally checks for errors. However, due to a bug in the firmware, the system incorrectly accepted a faulty configuration change. 4 minutes after the change was applied, both forwarding engines were reported unreachable by the routing engine. The issue was promptly detected and reported by our monitoring systems. An automatic rollback occurred one minute following the failure due to a lack of manual confirmation, which restored both routing engines and forwarding engines. Safety mechanisms have worked as expected and resulted in a quick recovery completed by 7:27 PM. To preemptively harden our infrastructure against similar issues in the future, we already planned to transition from a redundant routing chassis to completely isolated devices, ensuring full redundancy even in case of such failures. This transition is scheduled for April/May during a planned maintenance. We appreciate your understanding and patience as we work to enhance our systems' reliability.

Your product runs on more than LUMASERV

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →