Is Vetta Online down?

Some customers may be experiencing issues sending emails via our cp1 mail server. Engineers are investigating.

Issue has been identified as the recent ICANN KSK rollover, affecting DNSSEC validation being used as part of internal DNS recursion, which has not automatically moved to the new KSK keys. Remediation works are in progress and we expect functionality to be restored shortly. We apologise for any inconvenience caused.

Remediation works have been completed and outbound mail is now flowing, with queued emails clearing. Incident moving to Monitoring state to ensure service restoration.

Resolved

In order to resolve a sporadic route performance issue impacting some connections served from our Wellington PoPs to some destinations, our team will be performing route table updates to Wellington routing engines overnight tonight between 11pm-5am. While no impact is expected, there may be some latency irregularities to destinations while routes re-propagate.

During the period of ~05:30~06:30 this morning, a DDoS attack took place to our network, targeted at a downstream customer. Our networks team was mobilised and have since mitigated against this attack which may have caused high latency, jitter, and/or loss. Normal service has been restored. We apologies for the interruption and inconvenience caused during this period.

Resolved

During the period of ~2:00am ~ 8:35am this morning (17/09), some cloud compute customers with servers in Auckland and public IPv4 addressing contained within 103.85.28.0/24 subnet may have experienced degraded, or no IPv4 connectivity to these IP addresses. This was caused by this specific route not being distributed within our IGP following planned network maintenance overnight. Root cause investigation is underway, as well as increased monitoring of this IP subnet to proactively alert us to this issue in future. We apologise for any inconvenience caused. This issue was opened retrospectively.

Resolved

Some customers may be experiencing reduced connectivity, or higher latency to some destinations as a result of an incident with a core network element and subsequent routing path changes. Engineering teams are involved and investigating. Updates to follow.

Current investigations are showing the loss of two core network paths out of one of our core Auckland network PoPs. Normal traffic levels have moved to alternate and less-preferable traffic paths. Investigations are continuing.

Disrupted paths have been restored by the fibre vendor, and traffic has reconverged back to normal network paths. Our team is continuing to monitor the situation closely.

Primary paths have remained stable with normal traffic flow and this incident is now resolved. Our infrastructure team will continue communication with appropriate teams of the fibre vendors involved to identify root cause. We apologise for any inconvenience caused while traffic moved to alternate paths during this incident.