UPDATE: Issue with storage at our Cloud platforms
Posted by Victor Neo on 03 April 2019 06:07 PM
(1) There was a recent hypervisor bug in the virtualization infrastructure running ReadySpace Cloud Servers.
(2) Currently, there are 2 hypervisor versions v6 and v7, and your VM is running on either or.
(3) The issue is v6 VMs lost connectivity to storage network all of the sudden.
(4) So all VMs that are v6 immediately became inaccessible.
What are the issues?
(1) v6 and v7 are able to co-exist within the same storage cluster, with certain limitations.
(2) Since v6 hypervisor disks are blocked from accessing storage network virtual machines under this version are offline. The data within v6 cluster disks took some time to replicate/synchronize to available cluster disks.
What we did to resolve?
(1) Increase the storage cluster so that v7 nodes are enabled to accommodate v6 VMs for the failover process.
(2) Contact software vendor for further investigation and verified due to hypervisor bug.
What is the status now?
(1) Storage data are back to normal and customers using v7 VMs should have no problems, however, we are investigating on some v7 VMs intermittent connectivity issue
(2) Customers who are using v6 VMs are still offline because we will need to upgrade them to v7 ASAP.
(3) Due to the complexity of the issue, we are working on a workaround to bring up offline VMs as soon as we can while waiting for vendor permanent fix.