Restore disconnected node
From MyWiki
We had an outage in the building which led to all sort of strange things (loss of network, sporadic disappearances of servers, etc)
One of the issues was outage of on three ESX nodes, thankfully, not instantaneous which allowed us to work through the issues and restore the service back to normal. But one of the nodes was still disconnected as far as vCenter was concerned even though we could SSH into it without a problem.
We couldn't migrate VMs out of it though. We couldn't get to the VMs neither over the network nor via console. Catch 22.
Since the VMs were on shared iSCSI storage, we decided to kill the problem node and get the VMs up on another node later. Not something you'd like to do, but there seemed to be no other choice left.
The problem node 10.33.33.15, good running node is 10.33.33.18. Shared storage is iSCSI LUN hosted on a QNAP NAS appliance.
So, 10.33.33.15 goes down, but that doesn't change the fact that the node and all the VMs are marked disconnected in vCenter and we can do nothing with it there.
SSH to 10.33.33.18 and check the storage available:
~ # df -h Filesystem Size Used Available Use% Mounted on visorfs 1.4G 366.5M 1.1G 25% / vfat 4.0G 2.8M 4.0G 0% /vmfs/volumes/5241a93f-b7dd45ec-e1d9-5cf3fc09c4fe vfat 249.7M 115.0M 134.8M 46% /vmfs/volumes/a8622371-f21c2f29-e927-779374556003 vfat 249.7M 4.0k 249.7M 0% /vmfs/volumes/693e6c7c-feec42d1-cfdc-cf9f0d4fc01e vfat 285.9M 145.3M 140.6M 51% /vmfs/volumes/f57412e8-354f182d-35e2-275460791eb0 vmfs3 403.0G 563.0M 402.5G 0% /vmfs/volumes/5241a95d-1704ff26-2d0b-5cf3fc09c4fe vmfs3 1.9T 1.1T 891.0G 55% /vmfs/volumes/517ff6d9-b7dec161-a899-001a64792b78
The very last line is the iSCSI LUN we were looking for. And here my VMs:
~ # ls /vmfs/volumes/517ff6d9-b7dec161-a899-001a64792b78/ EXCALIBUR FRCBCERTIF FRCBZABBIX1 SMDEVWEBTEST01_new FR-C-WDS-01 FRCBDC2 ISO SMLPXWIKI FR1-C-PS-01.smartandco.com FRCBSCCM SEVEN_64_EN fr1-c-dc-01.smartandco.com FR1-C-SAGE30 FRCBSOPHOS SEVEN_64_FR FR1-CORP-AdminC FRCBVCENTER SMAPPFILES FRCBBACKUP01 FRCBWSUS SMARTNAGIOS
So far, so good.
Let's get the first one over to this node:
~ # vim-cmd solo/registervm /vmfs/volumes/517ff6d9-b7dec161-a899-001a64792b78/SMAPPFILES/SMAPPFILES.vmx 320 ~ # vim-cmd vmsvc/getallvms Vmid Name File Guest OS Version Annotation 320 SMAPPFILES [Frcbnas3_Iscsi] SMAPPFILES/SMAPPFILES.vmx debian5_64Guest vmx-07
Continue on with the rest of the VMs until I get them all registered on this node:
~ # vim-cmd solo/registervm /vmfs/volumes/517ff6d9-b7dec161-a899-001a64792b78/SMLPXWIKI/SMLPXWIKI2.vmx 560 ~ # vim-cmd vmsvc/getallvms Vmid Name File Guest OS Version Annotation 448 SMAPPFILES [Frcbnas3_Iscsi] SMAPPFILES/SMAPPFILES.vmx debian5_64Guest vmx-07 464 EXCALIBUR [Frcbnas3_Iscsi] EXCALIBUR/EXCALIBUR.vmx winNetStandardGuest vmx-07 480 FRCBBACKUP01 [Frcbnas3_Iscsi] FRCBBACKUP01/FRCBBACKUP01.vmx windows7Server64Guest vmx-07 496 FRCBCERTIF [Frcbnas3_Iscsi] FRCBCERTIF/FRCBCERTIF.vmx windows7Server64Guest vmx-07 512 FRCBDC2 [Frcbnas3_Iscsi] FRCBDC2/FRCBDC2.vmx winNetStandard64Guest vmx-07 528 FRCBWSUS [Frcbnas3_Iscsi] FRCBWSUS/FRCBWSUS.vmx windows7Server64Guest vmx-07 544 SMARTNAGIOS [Frcbnas3_Iscsi] SMARTNAGIOS/SMARTNAGIOS.vmx rhel5Guest vmx-07 560 SMLPXWIKI [Frcbnas3_Iscsi] SMLPXWIKI/SMLPXWIKI2.vmx winNetStandardGuest vmx-07
All was good and we started those that we needed now on the good node (10.33.33.18) and started to recover the failed 10.33.33.15 which stubbornly didn't want to boot up now. Ended up powering it down, pulling out power cables and after waiting for a minute bringing it back up. It did.
But, the VMs we so nicely re-registered on 10.33.33.18 are still registered on 10.33.33.15 and when we decided to reconnect it in vCenter we saw the whole list of it there, with vCenter saying that it's going to add it to the cluster. Nice, but not what we needed.
SSH into 10.33.33.15 to un-register the VMs before re-connecting the node back into the cluster:
Here is the whole list, but note that the VMID is not the same as it now has on 10.33.33.18
~ # vim-cmd vmsvc/getallvms Vmid Name File Guest OS Version Annotation 128 FRCBDC2 [Frcbnas3_Iscsi] FRCBDC2/FRCBDC2.vmx winNetStandard64Guest vmx-07 224 FR1-C-SCCM-01 [Frcbnas3_Iscsi] FRCBSCCM/FRCBSCCM.vmx windows7Server64Guest vmx-07 272 SMLPXWIKI [Frcbnas3_Iscsi] SMLPXWIKI/SMLPXWIKI2.vmx winNetStandardGuest vmx-07 400 SMARTNAGIOS [Frcbnas3_Iscsi] SMARTNAGIOS/SMARTNAGIOS.vmx rhel5Guest vmx-07 480 FRCBCERTIF [Frcbnas3_Iscsi] FRCBCERTIF/FRCBCERTIF.vmx windows7Server64Guest vmx-07 496 FRCBBACKUP01 [Frcbnas3_Iscsi] FRCBBACKUP01/FRCBBACKUP01.vmx windows7Server64Guest vmx-07 512 FRCBWSUS [Frcbnas3_Iscsi] FRCBWSUS/FRCBWSUS.vmx windows7Server64Guest vmx-07 528 FR1-C-SAGE30 [Frcbnas3_Iscsi] FR1-C-SAGE30/FR1-C-SAGE30.vmx windows7Server64Guest vmx-07 544 EXCALIBUR [Frcbnas3_Iscsi] EXCALIBUR/EXCALIBUR.vmx winNetStandardGuest vmx-07 64 SMAPPFILES [Frcbnas3_Iscsi] SMAPPFILES/SMAPPFILES.vmx debian5_64Guest vmx-07
Let's take a shot at one of the VMs - SMARTNAGIOS:
~ # vim-cmd vmsvc/unregister 400 ~ # vim-cmd vmsvc/getallvms Vmid Name File Guest OS Version Annotation 128 FRCBDC2 [Frcbnas3_Iscsi] FRCBDC2/FRCBDC2.vmx winNetStandard64Guest vmx-07 224 FR1-C-SCCM-01 [Frcbnas3_Iscsi] FRCBSCCM/FRCBSCCM.vmx windows7Server64Guest vmx-07 272 SMLPXWIKI [Frcbnas3_Iscsi] SMLPXWIKI/SMLPXWIKI2.vmx winNetStandardGuest vmx-07 480 FRCBCERTIF [Frcbnas3_Iscsi] FRCBCERTIF/FRCBCERTIF.vmx windows7Server64Guest vmx-07 496 FRCBBACKUP01 [Frcbnas3_Iscsi] FRCBBACKUP01/FRCBBACKUP01.vmx windows7Server64Guest vmx-07 512 FRCBWSUS [Frcbnas3_Iscsi] FRCBWSUS/FRCBWSUS.vmx windows7Server64Guest vmx-07 528 FR1-C-SAGE30 [Frcbnas3_Iscsi] FR1-C-SAGE30/FR1-C-SAGE30.vmx windows7Server64Guest vmx-07 544 EXCALIBUR [Frcbnas3_Iscsi] EXCALIBUR/EXCALIBUR.vmx winNetStandardGuest vmx-07 64 SMAPPFILES [Frcbnas3_Iscsi] SMAPPFILES/SMAPPFILES.vmx debian5_64Guest vmx-07
Carry on and unregister the rest of it:
~ # vim-cmd vmsvc/unregister 64 ~ # vim-cmd vmsvc/unregister 544 ~ # vim-cmd vmsvc/unregister 528 ~ # vim-cmd vmsvc/unregister 512 ~ # vim-cmd vmsvc/unregister 496 ~ # vim-cmd vmsvc/unregister 480 ~ # vim-cmd vmsvc/unregister 272 ~ # vim-cmd vmsvc/unregister 224 ~ # vim-cmd vmsvc/unregister 128 ~ # vim-cmd vmsvc/getallvms Vmid Name File Guest OS Version Annotation
Now we can re-connect the node to the cluster and wait when DRS kicks in and move something on to it. Which it later did.
Problem solved :-)
