VMware Virtual SAN Network Design Guide v2.0 is just released

VMware has just released the document that covers network aspects of Virtual SAN design. 

Actually, the guide has been re-released as there was v1.0 before (hence the new one goes under v2.0), but as far as I am aware it was removed from the VMware web site due to some inaccuracies in it. So for a while people lacked validated design information on one of the key aspect of VSAN setup. I remember there were quite a few discussions on how to provide network redundancy and load-balancing for VSAN traffic and nobody could get a formal answer.

The guide is very comprehensive and even provides mutlicast configuration examples on Cisco and Brocade switches.

You can check the guide here

Check out the new VSAN 6.2 Hands-On-Lab

VMware has just released new "HOL-SDC-1608 What's New with Virtual SAN 6.2" Hands-On-Lab

It covers all new functionality the VMware brought in new VSAN 6.2, e.g. compression and deduplication on AF, new SPBM settings, new ESXCLI VSAN namespace, etc.

However, the HOL assumes you have basic knowledge of VSAN. For instance, networking design isn't covered here. 

Interestingly, the networking configuration in this HOL still contains 2  VSAN VMK interfaces whereas generally it is recommended to have only 1 VMK and provide HA by NIC teaming and Load Balancing with LACP.

Even though I have VSAN 6.2 deployed in my home lab I still skimmed through the VSAN 6.2 HOL and I can tell it is pretty useful as I have learnt something new.

You can find it here - VSAN 6.2 HOL

Migration from Windows vCenter 5.5 to vCenter Server Appliance 6.2 - Part 3 - Upgrade ESXi hosts with ESXCLI

My servers have ILO , but that's not working good as it is based on Java - all kinds of problems with java in browsers, especially on Mac. So I usually avoid using it even though I pretty often used it before to mount ISO on servers over the network for ESXi installation and upgrade.

Given my issues with ILO I opted for an easier upgrade option - using powerful esxcli command
1. Upload image to the datastore


2. Check the profiles list in your depot file




3. You can check the each profile's details

That will show you the VIBs it includes and even the corresponding KB



4. Once you choose required profile just run the following command




As you can see the host was upgraded successfully, but it has to be restarted.

How to deploy VSAN 6.2 Witness Virtual Appliance to VMware Fusion 8.1

Since I moved to vSphere 6 U2 I started to plan my VSAN deployment. The problem I have is that there are only two physical servers in my home lab and I have no physical server to host VSAN Witness appliance. I am trying to make it run on my Synology DS415+ (crazy, but seems to be doable), but I need to upgrade RAM on Synology first.

As a temp solution I thought I could run VSAN 6.2 Witness on my Mac. William Lam has already posted a great post on how to Deploy and Run VSAN 6.1 Witness in VMware Workstation/Fusion, but I faced a small issue when following it. 

First it warned me that there is OVF specification and hardware compliance mismatch.  

And after that it failed to proceed with error "Line 821: Unexpected element 'Propery'"

Line 821 referred to the line we have to adjust in Step 3 as per William's guideline. I don't really know whether OVF file for VSAN 6.1 Witness was different or it is a newer version of VMware Fusion that can't parse the OVF file, but here is what I did to make it work. 

Just a small note - all credits for the following procedure go to William Lam. I only adjusted a couple of steps in his procedure to make it work for VSAN 6.2 Virtual Witness Appliance running in VMware Fusion 8.1.

1. Extract content of the OVA file using VMware OVF tool.

/Applications/VMware\ OVF\ Tool/ovftool VMware-VirtualSAN-Witness-6.0.0.update02-3620759.ova VMware-VirtualSAN-Witness-6.0.0.update02-3620759.ovf

2. Create a copy of OVF file in case something goes wrong. It is a quicker option than extracting OVA content again

3. Open OVF (you can even use default textEdit utility) and adjust the appliance size by moving the text marked in Red

<Configuration off:default="true" off:id="normal">

to the following string

<Configuration off:default="true" off:id="tiny">

3. Go to line 821 and delete the following text - marked in the screenshot

Btw, even though textEdit doesn't show line numbers you can still press Cmd+L and type the line number you want to go to. 

You can save and close OVF file now.

4. Create new SHA1 checksum for updated OVF file

openssl sha1 VMware-VirtualSAN-Witness-6.0.0.update02-3620759.ovf

5. Update the OVF file checksum in manifest file

6. Now you can import VSAN 6.2 Witness, but don't press Finish yet.

7. Get to the VM's locations and open the package to get to the .VMX file

8. Open the .VMX file - again, textEdit works just fine, add the following code and replace the password

guestinfo.ovfEnv = "<?xml version='1.0' encoding='UTF-8'?><Environment xmlns='http://schemas.dmtf.org/ovf/environment/1' xmlns:oe='http://schemas.dmtf.org/ovf/environment/1'><PropertySection><Property oe:key='vsan.witness.root.passwd' oe:value='Password123'/></PropertySection></Environment>"

9. Once you save the file you can run the Witness Appliance.

Migration from Windows vCenter 5.5 to vCenter Server Appliance 6.2 - Part 4 - Moving to external PSC

We got to the last step of our short journey. In this post I will be deploying new PSC appliance and then will reconfigure my vCSA to use external PSC instead of embedded once. 
For the lab of my size Embedded PSC is a preferable option, however I am going to deploy another vCSA and a couple of nested ESXi servers later to be able to test more interestings scenarios in the lab. And for that I will need my vCSA to use External PSC.

1. Create a DNS A record for new PSC.

That's an official prerequisite when deploying new PSC or vCSA. Without DNS record the installation process will fail. 

2. Deploy new PSC

2. Select Datacenter

3. Choose Deployment option - link to PSC Descision tree

Btw, there is a great PSC Deployment decision tree which helps you to decide on the best PSC deployment option 

4. Join to existing SSO domain

5. Choose Site

6. PSC comes in one size only

7. Select the datastore for PSC

8. Configure Networking

9. Review Summary and Click Finish

10. SSH to the new appliance and check the replication status

You don't really want to repoint you vCSA to external PSC that failed to replicate current configuration from the embedded PSC or has some issues with services health 

11. Check the new PSC status in the Web client

12. Join new PSC to the domain 

I almost overlooked this aspect, but I accidentaly remembered that I had read something about it in William Lam's blog

13. Repoint vCenter to new external PSC 

and we are done here. The goal is achieved. 

Now I can get to configure and test new VSAN 6.2 and plenty of other new features of vSphere 6.

Migration from Windows vCenter 5.5 to vCenter Server Appliance 6.2 - Part 2 - Upgrade vCSA 5.5 to vCSA 6.0 U2

Interrestingly, the process of vCSA upgrade is similar to vCenter Server to vCSA migration. 

First new appliance is deployed and then data is copied over from the old appliance. That is actually a significantly safer of option of upgrade as at any point of the upgrade process you can roll back by simply powering back on your existing vCenter.

Before you start the upgrade procedure you need to complete the following:

1. Install Integration Plug-In from the vCSA ISO file

2. Create temp Ephermal PortGroup. 

This will be used as a temp network by new vCSA 6 to connect to the current vCSA to copy data across. Once the copy is over the new vCSA will take network settings of the old vCSA and this network connection will be dropped. Make sure new vCSA can talk to existing vCSA from this temp network.

Once you complete these prerequisites download and mount vCSA ISO file and run open vcsa-setup.html


1. Press Upgrade button

2. Confirm your current vCenter version allows to upgrade to vCenter 6.0


3. Accept EULA


4. Provide FQDN and credentials for the ESXi server you want to deploy new vCSA to




5. Specify name for new vCSA - that's only the VM's name.
Optionally you can enable ssh on the appliance.




6. Enter details of vCSA 5.5 and the host where it resides




7. And I bumped into the first problem




8. Luckily I quicky found the solution in VMware Communities. 

The problem was in extra sequence in Postgres schema which had to be removed.

Run the following command in the vCSA shell and restart the upgrade procedure.

/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "drop sequence if exists vpx_host_cnx_seq cascade"


9. Select Appliance Size

10. Select datastore where you want to place new vCSA 6 U2

11. Configure temp network setting. 

I created a Temp portgroup on the same vlan where vCSA 5.5 resides and took advantage of my dhcp server

12. Confirm upgrade wizard settings

13. And the upgrade process failed with message 'Internal error occurs during export'.

I read vSphere Release notes and found similar error that suggested to use static IP address instead of DHCP config for temp network in step 11. Once I re-ran the upgarde wizard again with adjusted settings it all worked just fine

14. Then I spotted there was something wrong with vFRC config.

ESXi was reporting no disk space for vFRC.



But when I tried to remove the SSD and add it again the task failed with error that there are VMs using vFRC.

I checked VMs' settings, but it looked like these were not configured for vFRC even though they were.

So I had to shutdown VMs and re-add SSD disk as VFRC capacity which fixed the issue.

Migration from Windows vCenter 5.5 to vCenter Server Appliance 6.2 - Part 1 - Convert vCenter Server 5.5 to vCSA 5.5

I have been postponing too long to upgrade my home lab on vSphere 5.5 to vSphere 6.
I have used all kinds of reasons to postpone it - getting ready for VCAP5-DCA exam, testing StarWind, playing with vRA and vROps, but I am out of excuses now.

I have also thought it would be good to move from vCenter server to vCenter Appliance. It is quite obvious that VMware strongly pushes customers to vCSA and it is currently on par with vCenter server feature-wise. The only feature vCSA is missing now when comparing with vCenter Server is high availability, which can be partially provided with FT, but it has its own limitations. Definitely, vCenter Server on Windows Server Cluster looks a bit more reliable solution right now. Although, VMware has already let us sneak peak on HA features of future build of vCSA. Can't find this VMworld Session right now, but you can google it. 

Another thing I wanted to play with in vSphere 6 is Enhanced Linked Mode with 2 vCSAs. To be able to use ELM I need External Platform Services. My current vCenter DB is hosted on external SQL server.

Unfortunately, there is no supported or direct upgrade path from vCenter Server to vCenter Appliance yet, but it is still doable. What I am planning to do is not a proper solution for production environment, but works just fine to home lab/Dev/Stage environments.

Here is  high level overview of the entire process. If you are prefer a step-by-step procedure just follow the links below.

1. Convert vCenter Server 5.5 to vCSA 5.5 

I will be doing this using awesome and popular VMware Fling - VCS to VCVA Converter.
*It is not supported solution, but the fling has very positive feedback. I hope VMware will integrate it in the next vSphere build.

2. Upgrade vCSA 5.5 to vCSA 6.2

3. Upgrade ESXi hosts to the latest build with ESXCLI

4. Reconfigure vCSA with embedded PSC to vCSA with external PSC

In this post we will be migrating vCenter Server to vCenter Server Appliance. 

Here are the things I did prior running the conversion:

• Uninstalled and unregistered the following products to ensure clean and simple migration: 
• vSphere Replication 
• Infrastructure Navigator 
• vRealize Operations 
• Disabled vSphere Update Manager service 
• Downloaded the same build of vCSA appliance as my current vCenter
• Downloaded Fling appliance and checked all pre-requisites are met

You will need the following information during the conversion:

• SSO Administrator Password
• vCenter server local administrator password
• DB Server name and port
• vCenter DB name
• vCenter DB Username and password

If you run vCenter on Windows 2012 Server you will need to adjust WinRM security settings by executing the following commands as Administrator:

winrm quickconfig
winrm set winrm/config/service/auth @{Basic="true"}

winery set winrm/config/service @{AllowUnencrypted="true"} 

1. Deploy Converter Appliance

2. During the OVF Import task click Yes to the warning  saying this OS identifier is not supported and that it will be marked as 'Other Linux 64-bit'

3. Power on the converter

4. Deploy vCSA 

Use the same IP and hostname as of the current vCenter when importing OVF, but DO NOT POWER ON vCSA yet. 
Take a snapshot of it in case problems arise during the migration and you need to reset the appliance to a clean state.

5. Proceed with migration Wizard

Enter vCenter Server's IP address and Local Windows Administrator password. I tried domain admin account, but it didn't work

6.  Once the copy process from vCenter to Converter is over you will be asked to shut down current vCenter and power on new vCSA. 

Don't login to vCSA yet.

7. Open vCSA console and wait until you see the following screen.

For some reasons static IP configured in OVF deployment wizard of vCSA didn't apply. Instead it was taking DHCP address. So I had to create DHCP reservation with IP Address of vCenter Server.

Update: After I completed all steps of migration I found another workaround in Fling comments '' Boot of the vcsa : we encountered the problem of off parameters injection. OVF injection needs the vcenter for first boot, but as the address is the same as vcenter, only one shold be booted up at a time. We have here a problem of chicken and egg that we solved with first booting the VCSA with network disconnected, then shutdown the server and finaly, connect the vcsa network through ESXi connection.)

8. Accept SSH key

9. Enter default password for root account in vCSA

At this stage Converter appliance will start copying saved data to vCSA

10. Enter DNS suffix

11. Log in to vCSA and complete configuration of the appliance

12. Choose Custom configuration

13. Choose embedded DB type

14. Configure password for the embedded database

15. Enable AD integration

16. Once you are done you can switch back to Converter console and click OK

17. Accept Certificate

18. Enter SSO Admin password

19. Confirm domain name is correct

20. Enter vCenter DB details

Grab a cup of coffe and enjoy a break - the DB migration can take a while, especially if you chose to migrate stats,events and tasks.

An hour later I got database migration completed and could open Web Client from new vCSA, but faced another issue

I checked the vCSA network settings and noticed that it was still using DHCP and vCSA was still getting the correct IP address of old vCenter Server, but looks like it didn't accept the DNS server address from DHPC. Once I changed the IP Address config to static vCSA managed to reconnect both ESXi hosts

I also had to upgrade virtual hardware of vCSA to version 10 so that I could configure it to use vFlash Read Cache.

That completes the almost smooth migration from Windows vCenter Server to vCenter Server Appliance.