Practical lessons - Fault Tolerant VM

I have read quite a few official docs and different bloggers' posts about VMware FT, but once again I got evidence that practical knowledge is a best way of studying.

FT seems to be amazing feature, but the long list of requirements and restrictions can stop most of the VMware enthusiasts. Nevertheless, FT is one of the topic of VCAP-DCA blueprint, so I spent half a day playing in my lab with FT and trying to enable FT just for one VM. 

I wrote down some important lessons I learned about it, even though they seem very obvious you need to have some practical experience to remember them forever :)

• The identical CPU model are required - in my lab it didn't work on hosts that had Xeon 5550 and Xeon 5650
• The identical patch/firmware level is required - the same here, hosts had different firmware versions
• The VM Monitor mode has to be set to Use Intel VT-x/AMD-v for CPU virtualization and software for MMU Virtualization". - Yes, I have read that EPT and Large Pages are not supported for FT, but I didn't know it had to be switched off manually.
• VM disk can be converted to thick-eager-zeroed one only when VM is shutdown. - I was quite sure it was doable while VM was running
•  Memory reservation is set to the amount of configured memory. - You need to be very careful enabling FT if you use HA with HA Admission Control configured with hosts number to tolerate. This can impact the slot size of your HA cluster. 
• It is possible to configure Hyperthreading Core Sharing mode to None to provide slightly better performance for FT VM. - Actually, I am not really convinced that it provides better performance for FT enabled VM, but at least it will guarantee no other vCPU will be scheduled with FT VM vCPUs on the same physical CPU. My main idea was to make sure it is compatible with FT. 
• When you disable FT the secondary VM still stays in vSphere, but it powered off and disabled. All historical data about FT performance is kept.
• When you turn off FT the secondary VM is removed and all historical data is delel.
• RDM disks are not supported in FT VM.

I have also found some minor bug when you turn off FT. When you enable FT for VM it warns you about memory reservation and says it reservation will be maintained while FT is on. 

However, when I turned off FT the VM still kept memory reservation. Not a big deal, but it is just good to know you need to check VM settings after FT is switched off. 

There are a lot of other problems/issues you can face with when enabling FT, but my story was not that long :)

If you find this post useful please share it with any of the buttons below. 

My VCP 5 Exam experience

One year ago I passed VCP4 exam. That was a tough adventure. In August 2010 I attended my training on vSphere 4.1 and it was my very first experience with virtualization technologies. 2 weeks after the training I built my first vSphere farm on new HP Blade servers and carried out my first migration from VI 3.5 to vSphere 4.1. It was a hard time for me - zero real life experience, lack of knowledge and understanding how all these technologies work. However, by October 2010 I was ready to take my first exam for VMware. The final score of exam was just a little bit higher than the passing score, but I was still happy since I fell in love with vSphere and virtualization world.

This year I have decided to take an exam without any hands on experience with vSphere 5 for the simple reason that I don't have free time to build a proper lab and play around with new features. All my time currently is devoted to a preparation for a VCAP-DCA lab. Since this preparation has been lasting for last 6 month I can say now I have a solid understanding of main vSphere technologies. Also, I have designed and built few small to medium size vSphere farms and gained some real life experience with vSphere. 

So, I have started with reading all "vSphere 5: What's New" documentation. After that I decided to take a mock exam on VMware site and then I found out that there were still some gaps in my vSphere 5 knowledge. Reading official documenation is not a big fun. Therefore I wanted to buy some good book that explains the new technologies in depth and, what is more important, provides examples of their practical implementations, which helps me a lot to memorize the material. Surprise-surprise, I found only one vSphere 5 book available on Amazon site, which is the great book by Duncan Epping and Frank Denneman, however it was not exactly what I wanted for VCP5 exam preparation. So, I had to go back to VCP 5 Blueprint and go through all links to official VMware documentation, which took me about a week. 

The exam was more difficult than I expected. I was actually a bit scared after I answered first 10 questions and wasn't sure if my answers were correct in 5 of them. I was surprised there were very few questions about configuration maximums and none about licensing. That was a bit frustrating as I have spent a lot of time in debates about new licensing model :)  Surprisingly, there were more  questions than I expected about vCenter appliance. I can definitely say this exam requires more real life experience with vSphere than VCP4. Good thing is that probably about 80% of exam material is inherited from VCP4 exam, therefore, if you feel confident in your expertise in vSphere 4 it shouldn't take you more than a week to go through blueprint briefly, focusing only on new features.  Additionally, I checked some popular virtualization blogs to get alternative view or interpretation of the particular feature and I really like reading implementation examples' stories.  

A day before exam I looked for more mock exams on the Internet and found some good links here. I would say all of them are not as difficult as the real one, but at least you can find out what your gaps in vSphere 5 knowledge are and gain some confidence in yourself :)

Now I can go back to my VCAP-DCA lab. I would like to wish a good luck for this exam to all of you who has the same passion for virtualization technologies as I have. :)

If you find this post useful please share it with any of the buttons below. 

Storage IO Control and funny licensing issue

Today I have been finalizing the new vSphere installation. Basically, I have already migrated all VMs from old infrastructure to newly built one and was doing kind of fine tuning, which includes setting up monitoring, logging, etc. 

One of the task was reviewing what I still remember about SIOC and whether there are any objections of enabling it on new vSphere.  After reading couple of useful VMware docs (1, 2) and checking performance stats in vCenter I thouhgt it might be useful to have it enabled. 

However, when I tried to enable SIOC on the first storage I faced the following error "The vSphere Enterprise Plus License for host esxi.company.local does not include SIOC. Please update your license." I was really confused. This error didn't make any sense.

I checked Licensed Features of all hosts in the new vSphere Cluster - all of them had Storage IO Control listed as one of the allowed product feature. After 30 minutes of checking all possible settings I found out that old cluster still had all these VMFS datastores presented to it and for some reason another vSphere admin added test ESXi host into the old cluster. Since there were no available Enterprise Plus license available that guy used Enterpruse license, which obviously doesn't cover SIOC. 

The fix was quick and easy - unpresented VMFS LUNs off this test host and rescanned the storage configuration.

If you find this post useful please share it with any of the buttons below. 

Remote Tech Support Mode in ESXi 4.1 - unexpected feature

Just found out an interesting feature of ESXi Remote TSM. One might even call it a security risk.

Usually I utilize Remote TSM to get access to command line interface of our ESXi hosts. By default, SSH access is disabled on all hosts. So I have to open it temporarily, run commands I need and then close SSH.

However, today I forgot to close my Putty client with SSH session to the host.While checking vSphere I noticed yellow alarm sign on one of the host which was warning that Remote TSM was enabled. I immediately disabled it.

20 mins later I noticed my putty client is still connected to that host via SSH and i still had access to command line of the ESXi host. I thought there was a bug in vCenter and it probably didn't disable TSM. However, when I tried to open a new SSH session to the host Putty has failed to setup one. Be default TSM session never expires and this means you have to consider adjusting timeout value for TSM.

1. Go to the Advanced Settings of ESXi host
2. Locate "UserVars.TSMTimeOut" key
3. Adjust it to the desired value in seconds.


If you find this post useful please share it with any of the buttons below. 

Updating network and HBA drivers on ESXi 4.x

Recently I have been working on the installation and configuration of small size vSphere farm based on HP Blade servers and HP FlexFabric virtual connect. During the testing phase of the project I faced issue with SCSI commands' failures on one of the HBAs. Despite the fact it was quite a small infrastructure - 1 enclosure, 2 FlexFabric VCs, 4 blades, 2 Brocade SAN switches, 2 Cisco 6500 switches and 1 EVA 6400, it was not really easy to say what exactly was causing an issue. Moreover, it was my first experience with FlexFabric VC. So, I was a bit disappointed with the first results of my installation.
Two days later I found the root cause - it was faulty 8Gb FC SFP. While troubleshooting I had to check everything - configuration, firmware, drivers, etc,  and that is when I found out that I had no idea neither how to check ESXi drivers' version nor how to update them. Quick search on the internet didn't give me a good step-by-step procedure, so I had to go through vSphere Command-Line Interface Installation and Scripting Guide and thought it would be good to document the entire procedure for the future use.

Basically, you have 2 options - you can either use vMA or you can install vCLI on Windows computer. With vMA you will need to first copy drivers to vMA. I am a lazy person, so I temporarily opened Remote Tech Support (SSH) on each host and used my workstation to update drivers.

The first thing you need to start with is to check is a current driver version.

Networking Driver

1. In vSwtich properties check the Network Adapter driver currently used . In my case it is bnx2x.
   
   
   
   
   
2. Login into ESXi via SSH with root account.
3. Run the following command - esxupdate query --v | grep bnx2x. The driver version starts right after 400. In the screenshot you can see there are 2 driver versions. These are the old driver that was deployed during the ESXi installation and the newer one which was installed with one of the ESXi  firmware upgrades.  

HBA Driver

1. Login into your ESXi host via SSH.
2. Change directory to /proc/scsi/qlaxxx (QLogic based) or /proc/scsi/lpfcxxx (Emulex based)
3. You will find several files (one file per each SCSI HBA adapter), in my system there are just two of them

4. Looking into the content of these files you can get driver version.

Now we are done with collection of the information. The next step is the driver upgrade proccess itself. The upgrade steps are the same for networking and HBA drivers. First you need to look for the latest verion of drivers you need at the VMware Download Center. 

1. Put the ESXi server into the maintenance mode - either from GUI, or from vCLI using command vicfg-hostops.pl --server SERVERNAME --operation enter

2. From vCLI run the following command. Make sure you have put the correct server name and path.

•  vihostupdate --server ESX01 --scan --bundle c:\update\hba.zip

This will tell you which of the drivers in your zip files are applicable and still not installed  on your ESXi host.

3. Once the output of previous command confirms everything is fine run the next command:

•  vihostupdate --server ESX01 --update --bundle c:\update\hba.zip

4. Repeat steps 2 and 3 for networking drivers.

5. Reboot the host and exit maintenance mode.

I would really appreciate any comments about quality of my posts (including my English :) ) and any wrong conclustion or mistakes I could make here.

If you find this post useful please share it with any of the buttons below. 

Transparent Page Sharing - efficiency of disabling Large Pages

Finally I could find some free time to prove how efficient Transparent Page Sharing is. To do so I had to disable Large Page support on all 10 hosts of our vSphere 4.1.

Let me briefly describe our virtual server farm. We have 10 ESXi hosts with 2 Xeon 5650 CPUs in each host. 6 hosts are equiped with 96 Gb and 4 hosts - with 48 Gb, which gives us 768 Gb in total. Since all CPUs use Hardware Assisted MMU all ESXi hosts aggressively back virtual machines' RAM with Large Pages. Therefore, TPS won't kick in unless there is a memory contention on a host.

We run 165 virtual machines that have 493 Gb of assigned vRAM. According to Figure1 during the last week we have 488 Gb of consumed RAM and 33 Gb of shared RAM. When I just started my study on TPS I was very surprised to see some shared memory even though Large Pages were enabled. However, now I know that these 33 Gb are just zero pages that are shared when VM is powered on, without even being allocated in physical RAM of ESXi host.

Figure 1. Memory stats - Last week

In Figure 2 we can see that CPU usage is very low accros all cluster for the last week. According to VMware enabling Large Pages can be pretty effective with regard to CPU usage. VMware documents show about 25% lower CPU usage with LP enabled. Considering that we have had about 3% only of CPU usage I didn't see a problem to increase it to get benefits of TPS.

Figure 2. CPU stats - Last week

It took me a while to disable LP support on all hosts and then switch on and off maintenance mode on each host one by one to start taking advantage of TPS. In Figure 3 you can see memory stats for the period of time when I was still moving VMs around. The value of shared memory is not final yet, but you can see how fast ESXi hosts scan and share memory for VMs.

Figure 3. Memory stats - 1 hour after change

Since we disabled large pages support we should get higher rate of TLB misses which in turn should lead to more CPU cycles spent for walking through two page tables:

1. The VM's page table to find Virtual Address to Guest OS Physical Address mapping
2. Nested Page Talbe to find Guest Physical Address to Host Physical Address mapping

However, if you look at Figure 4 you won't see CPU usage increase. The spikes you can see there are mostly caused by masssive vmotions from host to host.

Figure 4. CPU stats - 1 hour after change

Unfortunately, I couldn't do all 10 hosts at once during working hours as there are some virtual MS Failover Clusters running in our vSphere and I needed to failover some cluster resources before moving cluster node off the ESXi host, therefore the last host was done only 5 hours later. You can see that the last host was done around 7-20 pm in Figure 5. After 5 hours since change has been implemented we already had 214 Gb of shared memory. Consumed memory decreased from 488 Gb to 307 Gb, thus saving us 180 Gb already. As you can see in Figure 6 there is still no CPU usage penalty due to disabled Large Pages.

Figure 5. Memory stats - 5 hours after change

Figure 6. CPU stats - 5 hours after change

In a couple of screenshots below you can see stats 24 hours since the change was done.

Figure 7. Memory stats - 24 hours after change

Figure 8. CPU stats - 24 hours after change

I have seen some posts on VMware communities where people were asking about the CPU impact with Large Pages disabled. The conclusion of this post can be a good answer to all those questions -  switching off the Large Pages saved us 210 Gb and CPU usage still stays very low.

With vSphere 5 release the benefits of TPS became not significant, but I guess there will be a lot of companies staying with vSphere 4.1 for a while. Also new vRAM entitlements made TPS a bit more useful.

Nevertheless, it is difficult to generalize performance impact of disabling Large Pages.  I wouldn't recommend to do so without proper test as TPS effeciency can significantly vary due to your vSphere specifics.

Update: I have been using Veeam Monitor to collect all these stats in Cluster view, but today I also checked vCenter performance stats and discovered that there is a difference in Shared and Consumed values. However, if I check Resources stats (which includes all VMs) in Veeam Monitor I get the same stats as I can see in vCenter. According to vCenter the Shared amount of memory is even higher - 280 Gb against 250 Gb in Veeam Monitor. That basically means more than 50% of physical memory savings!


If you find this post useful please share it with any of the buttons below. 

Multiple complaints forced VMware to change vRAM Entitlements

As far as I am aware right after VMware had started to receive tons of negative feedbacks regarding new licensing model of vSphere 5 they began reviewing the clients' current vRAM usage. I guess they questioned quite a lot of clients to compare the current amount of physical RAM and actual vRAM assigned and made a proper conclusions.

So here are the latest rumors and I personally truly believe they will come true. 

• vRAM entitlement per Enterprise and Enterprise Plus license is going to be doubled to 64GB and 96GB corresspondingly. 
• vRAM entitlement per Essentials and Essentials Plus license will be increased from 24GB to 32GB.
• Maxumum vRAM for Essentials /Essentials Plus will increased to 192GB
• Maximum 96GB per VM will be counted against your vRAM pool limit even though you might assign 1TB to this VM.

Honestly, I think it is very adequate response to all criticism VMware got for the last few weeks, but I am wondering - why VMware marketing guys couldn't foresee such reaction? Perhaps it shows that VMware was not fully aware about how their products were used in client companies. I guess now VMware will arrange such kind of resource usage reviews on a regular basis.  

Taken from here: http://derek858.blogspot.com/2011/07/impending-vmware-vsphere-50-license.html

Update: Official announcement of vRAM entitlement change is expected on 3rd of August.

Update 1: Official announcement is here - http://blogs.vmware.com/rethinkit/2011/08/changes-to-the-vram-licensing-model-introduced-on-july-12-2011.html

Here is comparison table to the changes that were announced on 12th of July.

Another nice thing to mention - vRAM usage will be calculated as an average amount used for last 12 month. So rare spiky raises in usage of vRAM will slightly increase average vRAM usage, but you will not need to pay for such spikes perpetually.   

If you find this post useful please share it with any of the buttons below.