16th August 2019

vps05.den01 VLAN Install Resulted in Network Connectivity Issues

[ORIGINAL ALERT 12:09] An anomaly was detected by our telemetry software, our technicians have been alerted to the anomaly and will begin investigating shortly. We will provide updates as they become available.

[UPDATE 12:10] A VLAN was not installed correctly during the normal install of a client causing the hypervisor to briefly lose connectivity to the network. We quickly rolled back the update and are investigating the root cause of why what normally is a standard procedure resulted in a brief network connectivity issue.

[UPDATE 12:24] The original network outage was caused by automation software attempting to install a VLAN to this hypervisor during an automated install but for some reason resulted in the new VLAN being set as the native VLAN port causing a network interruption. Our network engineer was logged into the distribution switch and was immediately paged within seconds of the detected downtime and was able to rollback the bad configuration in less than 60 seconds from the initial bad commit. We have disabled the python script associated with this automated behavior and will continue testing in the lab before attempting to bring back to production to determine where the bug is.