Sunday 6 September 2015

ESXi and Guest VM time sync - learning from mistake

Today I was browsing some interesting blogs while getting ready for VCAP5 exam and  stumbled upon the excellent post about time syncing in Guest VM.

The most interesting part of the post for me was the following:

"Even if you have your guests configured NOT to do periodic time syncs with VMware Tools, it will still force NTP to sync to the host on snapshot operations, suspend/resume, or vMotion."

That was pretty big surprise for me as I have always had all my VMs synced with NTP on the OS level. 

Here is the KB1189 the article refers to and the excerpt from it

"Unlike the normal periodic time sync, which can move the guest clock only forward, these special time syncs can move the guest clock either forward or backward."

I did a quick create/delete snapshot test and found that time in the VM would only sync forward. 

If the VM's time was ahead of the time of ESXi host taking/deleting VM snapshot or suspending/resuming VM will never sync its time backward. 

So after scratching my head and doing google-fu I found descent deep dive document on Timekeeping in Virtual Machines from VMware. And there it was:

"Note that all versions of VMware Tools make one-shot corrections of the virtual machine clock in certain cases (see the next section for details), independently of whether the periodic synchronization feature is on or off. The corrections in two of those cases do set the clock backward if needed: when the VMware Tools daemon starts (normally while the guest operating system is booting), and when a user toggles the periodic clock synchronization feature from off to on. "

In the following sections we find this paragraph

"VMware Tools automatically updates the guest operating system’s time to match the host operating system’s time in a few other cases in which the guest can be expected to have lost a large amount of time, even if periodic clock synchronization is turned off. To maintain a fictitious time, you must set the following options to FALSE. "

And then the document depicts these cases



As you can see most of these cases suggests that some amount of time was lost during one of the actions from the table above and therefore VMware Tools will sync VM's time only forward for them.

I know it is pretty small thing and most people configure NTP settings once and never get back to them again, but for me this discovery was a much bigger event. And here is why.

Half a year ago I had a job interview (which I presumably failed) and the funny thing is that I don't remember the name of the company, I don't remember the guys who interviewed me and asked a bunch of technical questions, but I remembered the question I couldn't answer.

I can't remember the exact wording, but it sounded like this "You get a call saying that some of the virtual servers that are members of corporate AD haven't their time synced with domain controllers. After initial investigation you find out that they went unsynced at about 10pm, about the same time when backup job for these servers started. Find the root cause of the problem". Well, the question already contained the hint, but I had no clue that the VMware Tools sync Guest VM's time even though the periodic time synchronisation is disabled. 


And all 6 months I periodically went back to that question which was stuck in my head as irritating as meat that got stuck between your teeth. Finally I can sleep well now.

So don't forget to sync your ESXi hosts with more than one NTP servers and make sure the NTP service starts automatically.


No comments:

Post a Comment