Transparent Page Sharing - efficiency of disabling Large Pages

Finally I could find some free time to prove how efficient Transparent Page Sharing is. To do so I had to disable Large Page support on all 10 hosts of our vSphere 4.1.

Let me briefly describe our virtual server farm. We have 10 ESXi hosts with 2 Xeon 5650 CPUs in each host. 6 hosts are equiped with 96 Gb and 4 hosts - with 48 Gb, which gives us 768 Gb in total. Since all CPUs use Hardware Assisted MMU all ESXi hosts aggressively back virtual machines' RAM with Large Pages. Therefore, TPS won't kick in unless there is a memory contention on a host.

We run 165 virtual machines that have 493 Gb of assigned vRAM. According to Figure1 during the last week we have 488 Gb of consumed RAM and 33 Gb of shared RAM. When I just started my study on TPS I was very surprised to see some shared memory even though Large Pages were enabled. However, now I know that these 33 Gb are just zero pages that are shared when VM is powered on, without even being allocated in physical RAM of ESXi host.

Figure 1. Memory stats - Last week

In Figure 2 we can see that CPU usage is very low accros all cluster for the last week. According to VMware enabling Large Pages can be pretty effective with regard to CPU usage. VMware documents show about 25% lower CPU usage with LP enabled. Considering that we have had about 3% only of CPU usage I didn't see a problem to increase it to get benefits of TPS.

Figure 2. CPU stats - Last week

It took me a while to disable LP support on all hosts and then switch on and off maintenance mode on each host one by one to start taking advantage of TPS. In Figure 3 you can see memory stats for the period of time when I was still moving VMs around. The value of shared memory is not final yet, but you can see how fast ESXi hosts scan and share memory for VMs.

Figure 3. Memory stats - 1 hour after change

Since we disabled large pages support we should get higher rate of TLB misses which in turn should lead to more CPU cycles spent for walking through two page tables:

1. The VM's page table to find Virtual Address to Guest OS Physical Address mapping
2. Nested Page Talbe to find Guest Physical Address to Host Physical Address mapping

However, if you look at Figure 4 you won't see CPU usage increase. The spikes you can see there are mostly caused by masssive vmotions from host to host.

Figure 4. CPU stats - 1 hour after change

Unfortunately, I couldn't do all 10 hosts at once during working hours as there are some virtual MS Failover Clusters running in our vSphere and I needed to failover some cluster resources before moving cluster node off the ESXi host, therefore the last host was done only 5 hours later. You can see that the last host was done around 7-20 pm in Figure 5. After 5 hours since change has been implemented we already had 214 Gb of shared memory. Consumed memory decreased from 488 Gb to 307 Gb, thus saving us 180 Gb already. As you can see in Figure 6 there is still no CPU usage penalty due to disabled Large Pages.

Figure 5. Memory stats - 5 hours after change

Figure 6. CPU stats - 5 hours after change

In a couple of screenshots below you can see stats 24 hours since the change was done.

Figure 7. Memory stats - 24 hours after change

Figure 8. CPU stats - 24 hours after change

I have seen some posts on VMware communities where people were asking about the CPU impact with Large Pages disabled. The conclusion of this post can be a good answer to all those questions -  switching off the Large Pages saved us 210 Gb and CPU usage still stays very low.

With vSphere 5 release the benefits of TPS became not significant, but I guess there will be a lot of companies staying with vSphere 4.1 for a while. Also new vRAM entitlements made TPS a bit more useful.

Nevertheless, it is difficult to generalize performance impact of disabling Large Pages.  I wouldn't recommend to do so without proper test as TPS effeciency can significantly vary due to your vSphere specifics.

Update: I have been using Veeam Monitor to collect all these stats in Cluster view, but today I also checked vCenter performance stats and discovered that there is a difference in Shared and Consumed values. However, if I check Resources stats (which includes all VMs) in Veeam Monitor I get the same stats as I can see in vCenter. According to vCenter the Shared amount of memory is even higher - 280 Gb against 250 Gb in Veeam Monitor. That basically means more than 50% of physical memory savings!


If you find this post useful please share it with any of the buttons below. 

Multiple complaints forced VMware to change vRAM Entitlements

As far as I am aware right after VMware had started to receive tons of negative feedbacks regarding new licensing model of vSphere 5 they began reviewing the clients' current vRAM usage. I guess they questioned quite a lot of clients to compare the current amount of physical RAM and actual vRAM assigned and made a proper conclusions.

So here are the latest rumors and I personally truly believe they will come true. 

• vRAM entitlement per Enterprise and Enterprise Plus license is going to be doubled to 64GB and 96GB corresspondingly. 
• vRAM entitlement per Essentials and Essentials Plus license will be increased from 24GB to 32GB.
• Maxumum vRAM for Essentials /Essentials Plus will increased to 192GB
• Maximum 96GB per VM will be counted against your vRAM pool limit even though you might assign 1TB to this VM.

Honestly, I think it is very adequate response to all criticism VMware got for the last few weeks, but I am wondering - why VMware marketing guys couldn't foresee such reaction? Perhaps it shows that VMware was not fully aware about how their products were used in client companies. I guess now VMware will arrange such kind of resource usage reviews on a regular basis.  

Taken from here: http://derek858.blogspot.com/2011/07/impending-vmware-vsphere-50-license.html

Update: Official announcement of vRAM entitlement change is expected on 3rd of August.

Update 1: Official announcement is here - http://blogs.vmware.com/rethinkit/2011/08/changes-to-the-vram-licensing-model-introduced-on-july-12-2011.html

Here is comparison table to the changes that were announced on 12th of July.

Another nice thing to mention - vRAM usage will be calculated as an average amount used for last 12 month. So rare spiky raises in usage of vRAM will slightly increase average vRAM usage, but you will not need to pay for such spikes perpetually.   

If you find this post useful please share it with any of the buttons below. 

vCenter Inventory Snapshot - Alternative way of resolving vCenter's SQL issues

Somewhere on upgrade path from VI 3.5 to vSphere 4.0 and finally to vSphere 4.1 we screwed up Performance stats in vCenter. I could still see real time data, but I couldn't select any historical data. Considering my intention to disable Large Pages on all cluster hosts and prove huge RAM savings with TPS I needed that historical data. Surely, the first website I visited was VMware and I immediately found KB which describes all symptoms I experienced. With my about zero SQL experience I followed entire procedure and it even seemed that I fixed the issue, at least I could select historical data options.

However, after checking it out for couple of days I found out that historical data is broken and discontinuous. Moreover, when I tried to use Veeam Monitor for VMware I couldn't get any CPU or Memory usage data on vCenter level.  That is when I remembered that recently in someone's blog I have spotted Inventory Snapshot product in VMware Labs.
Generally speaking, Inventory Snapshot is a bunch of PowerCLI commands that retrieve all essential data from vCenter, such as Datacenters, Host, VMs, HA&DRS Cluster settings, Resource Pools, Permissions, Roles, etc. You can check the whole list of inventory yourself. Once it collected all data from vCenter it generates PowerCLI code that can reproduce your entire environment either on new vCenter or on existing one if you managed to corrupt your SQL database. As you already understood it doesn't save performance data, but in my case it was ideal tool. With all vCenter SQL databases backed up I could play around as much as I wanted.

Here is a brief overview of steps you need to pass through to get my problem solved. However, you definitely want  to read Inventory Snapshot documentation because I am not inclined to reproduce official documentation :)

1. Stop vCenter service and backup its SQL database.
2. Bring vCenter service back online.
3. Run Inventory Snapshot and provide administrator's credentials
4. Once all inventory is retrieved and saved to .ps file you need to provide root's password for ESXi hosts. This is the weakest part of this product since all password are stored in plain text, although we need to remember that this is a lab product.
5. Stop vCenter again, run command line, navigate to C:\Program Files\VMware\Infrastructure\VirtualCenter\ and run vpxd.exe - b. It will fully reinitialize vCenter database, as if it was newly installed. At this step I had some warning about licenses, but seems like it was not important
6. Start vCenter
7. Run PowerCLI, login to vCenter and apply .ps file you saved before to vCenter. With 10 hosts, 4 Resource Pools and 190 VMs it took Inventory Snapshot about 20 mins to fully import back all objects.

So, now I have all types of performance graphs and it was a way simpler than going through VMware KB to fix something in SQL database.

In comments on VMware Lab site I saw people's complaints about missing permissions after they imported Inventory, but in my case I have got all permissions back correctly implemented.

Very Important to know!!! - The problem I had was that Inventory Snapshot couldn't recreate some part of  dvSwitch. Luckily, it was used only for my couple of test hosts. The error says "The host proxy switch associated to dvSwitch no longer exists in vCenter". Today is too late to start investigation. Tomorrow I will try to inform developers if it is not known bug yet. Anyway, it is still kind of beta version and you need to take all precaution measures before you start playing with it.

PS. Veeam Monitor still shows zero CPU and Memory usage. According to Veeam Community forum I need to ask their support for a fix. Why can't they just post it on the website if it is well known issue?


If you find this post useful please share it with any of the buttons below. 

Vlan Tagging and use cases of VLAN ID 4095

It was quite surprising for me to learn how useful VLAN 4095 can be, but let me start from the basics.

There are 3 main approches for vlan tagging - External Swich Tagging (EST), Virtual Switch Tagging (VST), and Virtual Guest Tagging (VGT).

External Swich Tagging

With this approach ESX host doesn't see any vlan tags. All of them are stripped off by external physical switch and then traffic is sent to corresponding physical port. Therefore, you will need one vmnic per vlan from the ESXi  perspective. 

I really don't know in which situations such config might be useful.

Virtual Switch Tagging

This is the most popular way of connecting vSphere to physical network. All traffic down to the vSwitch is tagged with vlan id. It is responsibility of vSwitch now to strip off the vlan tag and send packet to virtual machine in corresponding port group. Thereby, you can run all vlans over one vmnic, but I guess you would like to have another vmnic for redundancy. The same logic is implemented to the packet travelling from virtual machine to physical infrastructure. Packet is delivered to vSwitch and before it is sent to physical switch the packet is tagged with vlan id according to the port group memebership of originating virtual machine.

Virtual Guest Tagging

In some occasions you  might need to deliver traffic with vlan tags directly to VM and let your VM decide what to do with it. To achieve such functionality you need to assign this VM to port group with vlan id 4095 and configure Guest OS Nics with needed vlans. Interresingly, as soon as you put vlan id 4095 it is automatically changed to All (4095).

I didn't pay too much attention to reading about EST and VGT for the simple reasons it is not used at all in our vSphere farms and I don't want to spend time on something I will never have hands on experience with. However, when I questioned myself how I can sniff traffc on vSwitch google lead me directly to vlan 4095. 

Here are the two popular use cases for vlan 4095:

1. Traffic sniffing - from time to time you face networking problems and woud like to use some sniffing tool (or network protocol analyzer in case of Wireshark) to see what is going on behind the scene. Usually, you could enable promiscious mode on specific port group and use one of the VM in this port group to listen to the traffic. However, this leads to additional security risks as Promiscious mode lets all VMs of this port group to detect all frames passed on the vSwitch in allowed vlan. This also doesn't let you to sniff traffic from different vlans simultaneously. That's why you would prefer to use special port group with vlan 4095 and Promiscious mode enabled, where you can connect virtual adapter of VM you will use for traffic sniffing.  

2. IDS - Another good use case of vlan 4095 is to provide your virutal IDS with possibility to inspect all vSwich traffic. According to "VMware VSphere and Virtual Infrastructure Security: Securing the Virtual Environment" book this is quite a common scenario for virtual IDS placement. 

If you know more situations where vlan 4095 can be useful feel free to share in comments. I really like to learn vSphere features with real life examples. 

If you find this post useful please share it with any of the buttons below. 

vSphere 5 - Virtual Storage Appliance

In one of the comments to numerous blog articles I have been reading lately I noticed quite a sceptic opinion about vSphere 5 Virtual Storage Appliance (VSA) with regard to its capabilities and price. That was quite motivating to spend a day on reading about VSA and making my own opinion about it..

So, here is a short overview of VSA from vSphere admin.

The main goal of VSA is to provide SMB companies with shared storage without buying physical NAS or SAN itself. Instead VSA will use internal storage of your ESXi hosts to create shared NFS storage that will be presented as a single entity to all hosts in your vSphere. This is how it looks with 3 node configuration.

Unexpected benefit of SnS

Just found out that if you are going to buy Support and Subscription (SnS) contract you are entitled for free copy of SUSE Enterprise Linux System (SLES) for VMware. This also includes free subscription to all patches and updates for SLES. Moreover, you get free technical support as well.
I guess it is a good reason for me to start studying Linux.



If you find this post useful please share it with any of the buttons below. 

vSphere 5 licensing - what upgrade path to choose?

Found good licensing calculations of vSphere 5 upgrade based on quite a big production environment.
So i just followed this guy's advice and tried to make calculations for our production farm.

• 180 Virtual Machines with 427 GB of consumed RAM.
• 10 licensed hosts HP BL460 in 1 cluster 20 CPUs, 720 GB RAM total
• 20 Enterprice Edition licenses

If we want to keep vRAM equal to current physical RAM and stay with Enterprise edition once we upgrade to vSphere 5 we will need 720GB / 32GB = 23. Since we have support contract for another 2 years the upgrade of 20 current licenses will cost us zero. We will need to by 3 more licenses only, the pricing hasn't been changed. It will cost us $2,845x3=$8,535.

There is another, more interesting way to upgrade to vSphere 5. We can move to vSphere 5 for free and upgrade to Enterprise Plus edition for $685 per license. It will cost us $685x20=$13700, and we are fully safe with 48GB vRAM entitlement of Enterprise Plus. We will be able to use up to 48GBx20=960GB.

In a nutshel, we have 2 options:
1. Stay with Enteprise edition and upgrade to vSphere 5 for $8,535
2. Move to Enterprise Plus edition of vSphere 5 for $13700, but have some spare vRAM entitlement and all fantastic new features of vSphere 5.

It is almost obvious that option number 2 is a winner, $5000 is a not a big deal for such a great package.

The only information I lack is what happens to support contract once we upgrade to vSphere 5. Do we have new support for vSphere 5?  Our contract is still valid and it will be waste of money if we upgrade before it close to expiry. This information can definitely influence my primary decision. I will update post once I obtain it.

All these calculations and figures just prove my first thoughts about new vSphere 5 licensing model - it is beneficial and flexible for owners of big farms and Enterprise Plus edition licenses. It is aimed to big players, and I am afraid smaller virutalization customers can swing a bit to free MS Hyper-V solutions. It also seems like VMware decided to give up on SMB market and focus on large cloud providers.

Tthe value of main overcommitment technology TPS is significantly decreased with new licensing. The overall rate of memory overcommitment will also go down, although my assumption was that the resource overcommitment was one of the main drivers towards virtualization.

Another bad impact I can think of is that vsphere admins will try to size their VMs to fit vRAM pools and such approach can badly affect VMs performance, thus, undermining trust and confidentce in virtualization technologies.

Update 1: As I understood if you have Production support contract you are entitled for free upgrade to vSphere 5 and you will still have your support contract which will be transfered to vSphere 5 as well.  I guess it is time to revise your current support contracts and their expiry dates.

Update 2: Just read something really interesting for those whose Support&Subscription contracts are expired. Normally it would force you to buy vSphere 5 plus new SnS contract. However, the information below means you can save quite a few thousands of your currency on upgrade to vSphere 5 through renewing your expired SnS contracts.

Reinstatement Options for Customers with Inactive SnS Contracts:

• The applicable SnS fees for the current contract term
• Fees that would have been paid for the period of time that the customer’s SnS contract was not active
• A 20 percent fee on the sum of the fees in the preceding two

Update 3:  Seems like we need to wait till VMware vRAM entitlement change today and then recalculate the price of upgrade to vSphere 5.




If you find this post useful please share it with any of the buttons below.