Issue:
- Backups started failing for a VM at the point of taking a Snapshot. The day it was manually kicked-off and failed. It was tried again via another browser. The error reported by the backup software shortly after each backup was strarted was that the disk did not have a UUID.
- The server showed uptime of 200+ days, the VM datastore location was local. The drive showed healthy, the other two VMs on the datastore were not showing any signs of issues.
- Looking at the VM in the VCA via both the HTML5 and the older Flash web interfaces show no snapshots.
- Next we checked the Settings for the VM since the disk info was missing from the Summary tab. Clicked Edit Settings displays an error that the main disk 0 MB. If you try to set the size to the actual size (95 GiB, in this case), it tells you the maximum is about 7.9 GiB. This is much smaller than 95 GiB, and it also doesn't match the individual disk parts that combine to make up the one VMDK disk.
- Investigation of the host filesystem via SSH shows that there are several snapshots and the deltas are growing quickly in size. The snapshot files and the delta VMDK files all show current date/time stamps. The VM definitely does have an active snapshot.
e.g. # cd /vmfs/volumes/vmhost1localstor1/VMName/
Troubleshooting Steps Tried:
- We checked the host HTML vSphere client, and it showed no Snapshots. Making an another new snapshot completed, but it still did not show in the Snapshot Manager in the host vSphere client nor in the VCSA ones.
- Investigation of the host filesystem via SSH, displayed the new snapshot, as well.
- Performing a Delete All snapshots command via the host vSphere client did nothing.
- We then restarted the VCSA (vCenter). After the vSphere clients were available, we tried both the HTML5 and the Flash web clients again. Neither showed any snapshots.
- Next we started a restore of the VM from 3 days back to another host and another datastore. The one from 2 days back was too small, only 15 GB.
- Since we had a "good" backup from 3 days ago, we risked shutting the VM down and restarting it. No improvement. No snapshots visible in the vSphere web clients.
- We then put the host in maintenance mode, migrated all VMs but this one VM to other hosts in the cluster, and rebooted the host.
- After rebooting the host, and taking it out of Maintenance mode, we started the VM. To our surprise, the VM actually started up. (You'd think with no disk suposedly, it wouldn't boot. ) The VM showed no snapshots in the Snapshot manager. The Edit Settings still showed a 0 MB disk.
- We confirmed via SSH to the host, that the snapshots were actually still there along with their delta-vmdk files. We also confirmed that the actual VMX file displayed the one disk correctly pointing to its original VMDK and it's VMDK-flat files correctly.
What Actually Resolved the Issue:
- Since the server hadn't been upgraded to vSphere 6.5, we decided to attach the upgrade baseline to the host, and we performed the upgrade, and waited.
- Upon reboot, we found that the VM now displayed all the snapshots and if Edit Settings was clicked, the disk properly showed the 95 GiB size. We proceeded to do delete the most recent snapshot using the VCSA (Flash) vSphere client, since that is the client we were using to do the upgrade. (The Upgrade Manager tab of the VCSA HTML vSphere client has that 401 error which is supposed to be because you are not in it often enough, but for us, it goes 401 within a day or two of restarting the VCSA. Since we don't trust it, we use the Flash vSphere client for Update Manager.) The snapshot removal never displayed in the task. So we initiated it again, and waited.
- Giving up on the Flash vSphere client, we brought up the host vSphere web client and saw that the first snapshot was being removed and that it was almost done. As expected the second request for the same snapshot failed, as it should. Closing both of those clients, we opened the HTML5-based VCSA vsphere client and preformed a Delete All Snapshots and waited the couple hours for it to remove all the snapshots.
Solution:
If it's not too painful, just delete the VM and restore from backup.
If there have been too many updates, and it's too painful, then upgrade the ESXi host from vSphere 6.0 U2 to vSphere 6.5 Update (latest - 8294253). It worked great.
previous page
|