Validating NSX VTEP connectivity

This post was inspired by recent incident at the customer environment where VMs were experiencing networking issues due to MTU size misconfiguration on the TOR switches.

If you ever worked with NSX-V and Logical Switches you are aware that NSX configures VTEP vmnics with MTU equal to 1600 bytes. This allows to support VxLAN encapsulation.
However, between every two VTEP interfaces is an L2 or L3 networking device that is potentially not configured to support baby Jumbo frames (that's another name for 1600 bytes packets).

There are many posts explaining how to check MTU size and network connectivity between VTEP interfaces. It is a simple ping using esxcli:

esxcli network diag ping --netstack=vxlan --host vmknic_IP --df --size=1572

Now, imagine you have a small transport zones with 10 hosts and each host has 2 VTEP interfaces. 

You will need to run the esxcli command 360 times to validate all combinations of VTEP pairs. 

With 64 hosts the number of required ping tests reaches 16,128.  Well, that's obviously something that requires automation. 

Hopefully, the future versions of NSX will have this validation step as part of NSX Health Check. 

Meanwhile, we can take advantage of Powershell to make our VTEP validation test a bit easier.

I didn't spend much time writing the script and had only my home lab for a test, so it definitely may have some bugs. 

Here is the logic of the script:

• Connects to NSX/vCenter and validates that connection was established successfully
• Builds array of Transport Zones and Hosts
• Builds array of Hosts and their VTEPs
• Iterate through each TZ-Host-VTEP and ping all other VTEPs in the transport zone. This is a full-mesh test. 
• The script uses pings with 2 different sizes – 64 and 1572 bytes. The first allows to check for connectivity issues and the larger packet validates that MTU size is configured correctly along the path between two VTEPs.
• The results are displayed on the screen in real-time
• Two reports are produced for each transport zone:
• Summary - a table with Source Host, Destination Host and the test result
• Detailed - a table that contains Hosts, VTEP names and IP Addresses, test result for different packet sizes and the error message, if any. 

Script has been tested with vSphere 6.5 U1 and NSX 6.3.x

Update (9/11/2017) - the script was updated to work with ESXi 6.0 and 6.5 versions.

The following screenshot provides and example of successful tests:






This is an example of error messages when using packet size 1573




As you can see the script can detect different types of issues.

here is a couple of reports' screenshots

Summary Report

Detailed Report

Here is the script code

Feel free to provide feedback on any bugs you may encounter using this script. 

Updating configuration of NSX Controllers and Edge appliances

If you have been playing with NSX you may have noticed that you cannot edit settings of virtual appliances deployed by NSX, e.g. controllers or Edge appliances. That's how VMware want to ensure the best performance of NSX in your environment.  However, there might be cases when you still need to adjust some NSX appliances' settings.

In my case I needed to be able to change Memory Reservation settings. The thing is that all NSX appliances are deployed with 100% of memory reservation. My home lab grew up to almost 200Gb of RAM, but I still struggle with lack of memory especially when I run few nested deployments, each with its own NSX.

I am a big fan on PowerCLI so I tried to use Set-VMResourceConfiguration command let, but that attempt wasn't successful.

as you can see in the screenshot this method is disabled.

You can check all the methods disabled for VMs using this command

(get-vm VMname).ExtensionData.disabledmethod

As you can see the ReconfigVM_Task is in the list of disabled methods, which prevents any changes to the VM config.

There is a way to enable this method, but it can only be done through vSphere MOB, but I personally find it really confusing and not user friendly. And I had no clue how to automate this process. So, I gave up on this.

Then I thought there should be a way to change NSX appliances config through NSX RestAPI. And actually there is.

Here is how you can change the memory reservation of NSX edges using curl.  Update the values in bold before using.

1. Grab the NSX edge config and save it in XML file

curl -k -u 'username:password' -H "Content-Type: application/xml" -X GET https://nsxFQDN:443/api/4.0/edges/Edge-ID/appliances/highAvailabilityIndex  > XXX.xml

2. Update the memory reservation in xml file.

<memoryReservation>
<limit>-1</limit>
<reservation>YYY</reservation>
</memoryReservation>

3. Update the edge config

curl -k -u 'username:password' -H "Content-Type:application/xml" PUT https://nsxFQDN:443/api/4.0/edges/Edge-ID/appliances/highAvailabilityIndex -d "@XXX.xml"

As you can see you can change some settings of the Edge, but you cannot do the same with controllers. At least I couldn't find anything similar for controllers in NSX RestAPI guide. 

Also, it is not easy to automate.  

Here is an example of how you can use PowerCLI to automate RestAPI calls

And here what you can get from the output

From here you can update anything you need and change the config using similar PowerCLI function.

As you can see it is more time consuming way of doing things. and again, this is not applicable for NSX controllers.

So I thought I should go back to the original idea of enabling ReconfigVM_task method and started searching for instructions when I found out (once again) that William Lam has already done this. In this post he explains how you can disable vMotion for some of the VMs by disabling MigrateVM_task method. But the most amazing part of that post was that he created PowerCLI functions to enable/disable any methods without using vSphere MOB.

From here it was really easy to create the following script which changes the memory reservation on any VMs - whether they are deployed by NSX or not.

The script grabs all VMs with 100% of memory reservations and changes this value to 99%.  You can change this value to whatever you prefer. If ReconfigureVM method is disabled the script will re-enable it first. After the memory reservation is updated the script will change the ReconfigVM method back to disabled.

All you need to do is to update the vCenter name and credentials before you run the script.

Here is the example of the script output

A word of caution - this is not officially supported way of changing the settings of NSX appliances.  It works but it's at your own risk.

Creating replica seeds for vSphere Replication

I have known for a while that vSphere Replication allows to use replica seeds to significantly save time of initial sync.

This approach is recommended when there is not sufficient bandwidth between sites to complete replication in time. In this case it is recommended to create copies of the VM disks, transfer them to the destination site using external media, e.g. external USB hard drive. Once the files are copied to the target datastore vSphere Replication can be instructed to use them as replica seeds. The source and target disks will be scanned and only modified blocks of data will be transferred.

There is an issue with this approach. According to VMware documentation the virtual machine has to be powered off before creating disk copies of the original VM. In most environments this kind of action require Request for Change and it can take quite a while before this request approved.

As a workaround you can clone powered on VM, but the disks of the cloned VM will have new UUID. When vSphere Replication is instructed to use replica seeds it compares source and destination disks using two criteria - VMDK name and UUID. If one of them doesn't match in both disks you won't be able to configure vSphere Replication for this VM.

Therefore, I thought it is a nice opportunity to simplify process of creating replica seeds for vSphere Replication with no outage for virtual machines.

So, the whole process is quite simple:

1. Clone running VM. The cloned VM will need to have the same name to keep the disk names identical. Since VMs will have the same name they will need to be placed into different folders.

2. Run the script that will update the cloned VM's disk with original UUIDs.

Automating configuration of a scratch location with PowerCLI

Quite often the modern ESXi servers come with no local storage and ESXi is normally installed on SD card.

As per VMware KB1033696 the SD card can't be used to store scratch partition. The main purpose of the scratch partition is to store logging files and to provide space for vm-support output.

So, the normal practice is to use shared storage (VMFS/NFS) as a scratch location. The problem is that the configuration of the scratch location is not automated in the existing vSphere. So you have to manually create folder for each of the ESXi host and configure each ESXi host to use that folder.
This can be quite time-consuming and boring tasks when you have to do it for hundred of servers.

To make things worse Host Profiles do not let you configure scratch location too.

I had some time last week and thought it was a good chance to have fun with PowerCLI and automate the scratch configuration for ESXi hosts.

So here is overview of what the script does:

1. Connects to vCenter
2. Collects the list of ESXi hosts in the cluster. Very often storage is not shared across multiple compute clusters so I decided to use cluster, not a datacenter, as a configuration target.
3. Checks if there is a designated scratch folder for each of the clusters and creates if it doesn't exist
4. Checks if the ESXi host configured with scratch location and if it points to the right datastore and folder.
5. If ESXi is not configured yet or points to the wrong directory the correct setting will be applied.
6. Provides a list of the ESXi servers to be rebooted for the configuration change to take effect

There are a couple of thing you have to do before running the script:

• Identify the datastore to be used to store scratch folders
• In that datastore create a folder where the script will create a scratch folder per each host 

 The syntax is as following:

.\scratch.ps1 -vCenter vCenter_Name -cluster Cluster_Name -datastore Datastore_Name -folder Folder_Name

for example

.\scratch.ps1 -vCenter lab-vc-01.lab.local -cluster HA -datastore ISO -folder Scratch

* I had to add folder as input parameter because I couldn't make the script land into the correct folder with New-PSdrive commandlet

You can go even further by taking advantage of Windows Task Scheduler to run this script on a daily basis to ensure all servers are consistently configured.

Let me know how it worked for you.