Isolating vSphere Replication Traffic

One of the new great features of vSphere Replication 6 is traffic isolation, which significantly enhances security and facilitates QoS using Network I/O Control feature.

Even though TCP/IP stacks are not useful for moving vSphere Replication traffic to separate network it is not too difficult to achieve the same result using static routes.

In this post I will show the different types of vSphere Replication traffic flows and will explain how to achieve full isolation of the replication traffic from management network.

Automating configuration of a scratch location with PowerCLI

Quite often the modern ESXi servers come with no local storage and ESXi is normally installed on SD card.

As per VMware KB1033696 the SD card can't be used to store scratch partition. The main purpose of the scratch partition is to store logging files and to provide space for vm-support output.

So, the normal practice is to use shared storage (VMFS/NFS) as a scratch location. The problem is that the configuration of the scratch location is not automated in the existing vSphere. So you have to manually create folder for each of the ESXi host and configure each ESXi host to use that folder.
This can be quite time-consuming and boring tasks when you have to do it for hundred of servers.

To make things worse Host Profiles do not let you configure scratch location too.

I had some time last week and thought it was a good chance to have fun with PowerCLI and automate the scratch configuration for ESXi hosts.

So here is overview of what the script does:

1. Connects to vCenter
2. Collects the list of ESXi hosts in the cluster. Very often storage is not shared across multiple compute clusters so I decided to use cluster, not a datacenter, as a configuration target.
3. Checks if there is a designated scratch folder for each of the clusters and creates if it doesn't exist
4. Checks if the ESXi host configured with scratch location and if it points to the right datastore and folder.
5. If ESXi is not configured yet or points to the wrong directory the correct setting will be applied.
6. Provides a list of the ESXi servers to be rebooted for the configuration change to take effect

There are a couple of thing you have to do before running the script:

• Identify the datastore to be used to store scratch folders
• In that datastore create a folder where the script will create a scratch folder per each host 

 The syntax is as following:

.\scratch.ps1 -vCenter vCenter_Name -cluster Cluster_Name -datastore Datastore_Name -folder Folder_Name

for example

.\scratch.ps1 -vCenter lab-vc-01.lab.local -cluster HA -datastore ISO -folder Scratch

* I had to add folder as input parameter because I couldn't make the script land into the correct folder with New-PSdrive commandlet

You can go even further by taking advantage of Windows Task Scheduler to run this script on a daily basis to ensure all servers are consistently configured.

Let me know how it worked for you.

vSphere Distributed Switch and Nexus 1000v comparison

Choosing between VMware and Cisco virtual switch products is not an easy tasks as it includes not only side-by-side feature comparison, but also numerous aspects of duty separations, operational overhead, current skill set and expertise. And not all of them can be compared directly. 

Apart from all that it can be simply a political decision to a question "Who is going to manage virtual networks?". 

In this article I am trying to provide essential information on things to help you make the right decision for your infrastructure.

C# Client is dead, long live the Web Client

Yep, C# client will no longer be available in the next releases of vSphere.

VMware have been giving a pretty clear signals in the last major vSphere releases that C# client would be gone soon, but nobody knew when exactly it was going to happen. However, when SRM and VUM were moved to web client in vSphere 6 it was obvious C# client's days are numbered.

Probably it could have been done a couple of years ago, but first Web client wasn't good enough due to performance issues. It also lacked integration with other VMware solutions and there was no replacement of C# client for standalone ESXi hosts.

Today VMware have moved its plugins (SRM, VUM) to the Web client and other VMware partners are sailing in the same direction. It also presented new embedded HTML5 Host client for ESXi hosts, which has feature parity with C# client for standalone hosts.

Here are some of web client benefits compared to C# client:

• Scalability – WC handles more objects and more concurrent admin access
• Bookmarking URLs - WC allows you to quickly return to specific objects or views and share them with others (such as in a support ticket)
• Recent Objects - WC lets you navigate quickly between things you’re working on
• Work-In-Progress - WC lets you save your work and come back to it later, even from another computer!
• Enhanced Linked Mode – WC can call up all your inventory in one view
• Remembers user customizations to UI – WC enables column selections and widths on grids, portlets on summary pages
• Latest feature support – WC is the only interface to support all new features 

As a short-term goal I think VMware will be focusing on fixing Client Integration Plugin which causes most of the issues with Web client - people having issues with OVF import, browsing datastores. It also doesn't work on Mac.

The long term goal would be to have a single ultimate client for vSphere and ESXi hosts. That's what actually VMware is doing right now by trying to replace Flash Web client with HTML5. You can already have a preview of H5 Web Client for vSphere - it exists as a Fling.

It has to be noted that The C# client will be kept in all current platforms.

You can read the official announcement here and that's where you can leave your feedback. 

Why I prefer to disable vSphere HA Admission Control

According to vSphere Documentation "vCenter Server uses admission control to ensure that sufficient resources are available in a cluster to provide failover protection and to ensure that virtual machine resource reservations are respected."

Admission Control guarantees enough capacity is available for virtual machine failover and it works at 3 different levels - Host, Resource Pool, Cluster.  Only the Cluster level Admission Control is manageable and that's what I want to talk about.

Even though the idea behind Admission Control is pretty simple this mechanism is a complex topic and according to the best book about vSphere HA by Duncan Epping and Frank Denneman "Admission Control is more than likely the most misunderstood concept.. and because of this it is often disabled".

To achieve its goal the admission control can use the following actions:

• Restrict Powering on a virtual machine. 
• Disallow Migrating a virtual machine onto a host or into a cluster or resource pool. 
• Restrict Increasing the CPU or memory reservation of a virtual machine. 

In simple words vCenter uses Admission Control to have enough resources to power on VMs in case one or more of the hosts fail, but not to guarantee VMs have the same set of resources.

To achieve this goal Admission Control can use 3 different mechanisms. Let's discuss what diffuculties vSphere admin may have with each of them.

Host Failures Cluster Tolerates

Slot size will be the main factor defining when vCenter won't let you power on another VM. This admission control type gets skewed very easily by having one large VM with all RAM reserved.

Yes, you can define custom slot size, but deciding on the right size of the custom slot isn't a trivial task. If you set it too low you can't be sure you have enough reserved resources for host failure. If you set it too high you will end up with fewer number of slots you can use for your VMs. So you can end up with cluster with plenty of available resources, but won't be able to deploy new VM. Also, most of the vSphere environments constantly grow and change. therefore, slot size have to adjusted accordingly.

One can calculate average VM size and use these values for custom slot size. However, while vCenter will reserve enough slots to tolerate failure of configured number of hosts it won't guarantee that either of your host will have enough resources to power on your largest VM. 

HA Cluster is smart enough to ask DRS to shuffle VMs around to make space for your large business critical server. But, there is still no guarantee of successful restart of this VM. Moreover, we all know that DRS is part of vCenter. So if vCenter was located on the failed ESXi host there will be no DRS to take care of resource defragmentation in the cluster.

Percentage of Cluster Resources Reserved

This Admission Control seems to be smarter and more flexible than the first one. vCenter doesn't use fixed slot size any more. This is what most vSphere admins recommend to use, but nobody stresses out that in order to have this policy work you need to set CPU/RAM reservation per every single VM in the cluster.

So here is what vCenter does:

1. Calculate how much resources it has in the cluster 
2. Calculate total reserved virtual machines resources 
3. Calculate available resources by distracting the reserved resources from available resources. 
4. Ensure that there is more available resources than percentage of reserved resources for failover (that's the percentage you configure) 

As you can see it is not based on assigned resources to VM. This equation works with RESERVED resources only.

So if you need to end up with properly working admission control you need to assign reservation on each of the VMs. Otherwise, you may end up with situation like in the following screenshot where we you can see that the cluster runs almost 300 VMs, but Admission Control thinks you have 98% of Cluster Failover Capacity.

This is very common situation where people don't use reservations at all or use Resource Pool reservations (which are not taken into consideration by Admission Control). In this case Admission Control uses default 32MHz and Memory overhead in its calculation and your cluster will run out of resources way before admission control kicks in and prohibits powering on new VMs.

Well, you can say - let's use VM reservations then to make admission control work properly, but I have to disagree for the following reasons:

• Reserverd RAM can't be re-allocated to other VMs even if it's not used at all. That's valid for Windows VMs as they zeroize all RAM pages during the boot. Linux VMs don't touch RAM pages until they really want to use them, thus, these 'untouched' memory pages can be still re-allocated to other VMs. 
• It also leads to higher administrative and operational overhead. Imagine micro-managing correct reservations for thousand of VMs and then keeping these reservations up to date. 
• VMware Performance Best Practices recommend using Resource Pool Reservations instead of VM level reservations

On top of that you have to remember to change percentage every time you change number of hosts.  And if you have unbalanced cluster you need to have different percentage reservation for CPU and RAM.

This approach doesn't solve the problem of resource defragmentation either. DRS still needs to kick in if the host doesn't have enough resources to restart VM. And again, if vCenter is down there is no DRS. Which means there will be no defragmentation and some large critical VMs won't power on.

Dedicated Failover host

This is the least preferred option according to multiple blogs and books on HA Admission Control, but I strongly disagree with such opinion.

First of all it is very simple and doesn't require a lot of planning. it is pretty much 'You get what you see' 

The main reason why people recommend against using the dedicated failover hosts is that these hosts are not utilised until other hosts fail. Although that's actually applicable to other Admission Control mechanisms as you still have to reserve resources for failover - which means you keep those reserved GHz and GB unused. 

This approach doesn't have issues with resource defragmentation and doesn't need DRS. So if vCenter fails with failed host you still get your large VMs restarted.

This Admission Control type comes with some drawbacks:

• When you have hosts in this hot-standby mode the ratio of vCPU/pCPU gets worse as less physical CPUs participate in servicing vCPUs, which can impact CPU Ready time. 
• Another problem could be if this server is a part of VSAN cluster. That would be too much waste of resources. 

As you can see none of the Admission Controls is ideal. None of them fit all sizes. And each mechanism requires thorough planning, testing and regular revisions.

On the contrary to what Duncan and Frank said I prefer to disable HA Admission Control because I understand how it works . Instead, I think it is sufficient to have a good monitoring system in place to control vSphere utilisation level and see resource usage trends so that you can do capacity planning in advance. vRealise Operations suite is a good example of such monitoring system. 

I am not trying to say it is useless in all situations, but there were very few companies where I saw Admission Control correctly configured and looked after properly, and all of these few companies used Dedicated Failover host.

And the bigger problem was that people blindly relied on HA Admission Control even though it doesn't work correctly without per VM reservation or without right-sizing the slot size.

Another budget vSphere home lab post – Part 2

The first part of the post can be found here

Finally I got some time to write a second part about my home lab.

We'll start with some pictures of the build process, but if you find them boring just scroll down to the next section.

I just have to mention that the assembly process was very simple and straightforward even for the guy who built his last PC about 15 years ago. Took me only a couple of hours to get both servers powered on.

Another budget vSphere home lab post - Part 1

A slightly more than a year ago I bought the top specs Macbook Pro (i7, 16GB, 512 GB) being totally sure it will be sufficient to run simple nested vSphere Lab. Honestly speaking I didn’t think through the requirements and what exactly I want to run in it, I just wanted a new toy. When vSphere 6 Beta was released it was a big surprise for me that 16GB of RAM is just enough to meet minimum requirements for 2 hosts and vCenter. I am not even talking about attempts to fit vSphere Horizon or vCloud Director setup in my nested lab. I used one quite powerful server at work for a while, but good time has passed very fast and it was definitely time to come up with a new permanent solution.