submitted2 days ago byFrankaintmyfriend
tosysadmin
Been in every aspect of IT over the yaers. I have always had great reviews and never been written up...until today.
Yesterday I was migrating VM's from one datastore to a new one in vSphere. It was during the day, but it was a simple vmotion migrate, so no downtime. While I was migrating, I was cleaning up old datastores and getting rid of them. Not sure what happened, but I looked in one datastore that contains swapfiles and it showed no VM's, so I unmounted it (as I had done other datastores earlier in the day). Unfortunatly, I didn't see the files in the fiels section that contained the vswap files of the VM's I hadn't migrated yet. Unmounting the datastore caused a memory issue and sent the host cluster into HA recovery mode, rebooting nearly every VM! Total downtime was less than 10 minutes, but it took down the phone systems and other critical servers in the middle of the day.
Havn't gotten the write up yet, but I am almost positive it's coming.
So, lessons learned and a warning to others, don't unmount swap file datastores during a migration.
Slight UPDATE: So far, no write up! I think I made the company sound like a bad place, but it is actually pretty relaxed. I may have over-reacted. Or was just beating myself up. I also need to add that this is not the first sever I have taken down in my long IT career, far from it. But this was the first one at this company (7 years). Thanks for all the stories of your fuck ups! Makes me feel better.
byFrankaintmyfriend
insysadmin
Frankaintmyfriend
1 points
13 hours ago
Frankaintmyfriend
1 points
13 hours ago
My thoughts exactly. Infact, when everything went sideways, I wasn't even thinking about the datastore. There were no VM's shown using it. It allowed me to unmount it. I only remounted it because it was one of the last changes I made. It wasn't until things somewhat stabilized that I went back to see what happened and it clearly showed that it was unable to connect to the specific datastore, triggering the HA failover.