Is Quick NZ down?

In order to resolve a sporadic route performance issue impacting some connections served from our Wellington PoPs to some destinations, our team will be performing route table updates to Wellington routing engines overnight tonight between 11pm-5am. While no impact is expected, there may be some latency irregularities to destinations while routes re-propagate.

During the period of ~05:30~06:30 this morning, a DDoS attack took place to our network, targeted at a downstream customer. Our networks team was mobilised and have since mitigated against this attack which may have caused high latency, jitter, and/or loss. Normal service has been restored. We apologies for the interruption and inconvenience caused during this period.

Resolved

During the period of ~2:00am ~ 8:35am this morning (17/09), some cloud compute customers with servers in Auckland and public IPv4 addressing contained within 103.85.28.0/24 subnet may have experienced degraded, or no IPv4 connectivity to these IP addresses. This was caused by this specific route not being distributed within our IGP following planned network maintenance overnight. Root cause investigation is underway, as well as increased monitoring of this IP subnet to proactively alert us to this issue in future. We apologise for any inconvenience caused. This issue was opened retrospectively.

Resolved

Some customers may be experiencing reduced connectivity, or higher latency to some destinations as a result of an incident with a core network element and subsequent routing path changes. Engineering teams are involved and investigating. Updates to follow.

Current investigations are showing the loss of two core network paths out of one of our core Auckland network PoPs. Normal traffic levels have moved to alternate and less-preferable traffic paths. Investigations are continuing.

Disrupted paths have been restored by the fibre vendor, and traffic has reconverged back to normal network paths. Our team is continuing to monitor the situation closely.

Primary paths have remained stable with normal traffic flow and this incident is now resolved. Our infrastructure team will continue communication with appropriate teams of the fibre vendors involved to identify root cause. We apologise for any inconvenience caused while traffic moved to alternate paths during this incident.

A connectivity disruption is currently being experienced to some hosted customer cloud servers, and our team is investigating.

Connectivity has been restored and the team are monitoring.

Incident has been resolved and further preventative measures have been taken. Maintenance will be announced soon to complete changes in response to this incident and its subsequent root cause investigation. We apologise for any inconvenience caused.