Storage IO Control and funny licensing issue

Today I have been finalizing the new vSphere installation. Basically, I have already migrated all VMs from old infrastructure to newly built one and was doing kind of fine tuning, which includes setting up monitoring, logging, etc. 

One of the task was reviewing what I still remember about SIOC and whether there are any objections of enabling it on new vSphere.  After reading couple of useful VMware docs (1, 2) and checking performance stats in vCenter I thouhgt it might be useful to have it enabled. 

However, when I tried to enable SIOC on the first storage I faced the following error "The vSphere Enterprise Plus License for host esxi.company.local does not include SIOC. Please update your license." I was really confused. This error didn't make any sense.

I checked Licensed Features of all hosts in the new vSphere Cluster - all of them had Storage IO Control listed as one of the allowed product feature. After 30 minutes of checking all possible settings I found out that old cluster still had all these VMFS datastores presented to it and for some reason another vSphere admin added test ESXi host into the old cluster. Since there were no available Enterprise Plus license available that guy used Enterpruse license, which obviously doesn't cover SIOC. 

The fix was quick and easy - unpresented VMFS LUNs off this test host and rescanned the storage configuration.

If you find this post useful please share it with any of the buttons below. 

Remote Tech Support Mode in ESXi 4.1 - unexpected feature

Just found out an interesting feature of ESXi Remote TSM. One might even call it a security risk.

Usually I utilize Remote TSM to get access to command line interface of our ESXi hosts. By default, SSH access is disabled on all hosts. So I have to open it temporarily, run commands I need and then close SSH.

However, today I forgot to close my Putty client with SSH session to the host.While checking vSphere I noticed yellow alarm sign on one of the host which was warning that Remote TSM was enabled. I immediately disabled it.

20 mins later I noticed my putty client is still connected to that host via SSH and i still had access to command line of the ESXi host. I thought there was a bug in vCenter and it probably didn't disable TSM. However, when I tried to open a new SSH session to the host Putty has failed to setup one. Be default TSM session never expires and this means you have to consider adjusting timeout value for TSM.

1. Go to the Advanced Settings of ESXi host
2. Locate "UserVars.TSMTimeOut" key
3. Adjust it to the desired value in seconds.


If you find this post useful please share it with any of the buttons below. 

Updating network and HBA drivers on ESXi 4.x

Recently I have been working on the installation and configuration of small size vSphere farm based on HP Blade servers and HP FlexFabric virtual connect. During the testing phase of the project I faced issue with SCSI commands' failures on one of the HBAs. Despite the fact it was quite a small infrastructure - 1 enclosure, 2 FlexFabric VCs, 4 blades, 2 Brocade SAN switches, 2 Cisco 6500 switches and 1 EVA 6400, it was not really easy to say what exactly was causing an issue. Moreover, it was my first experience with FlexFabric VC. So, I was a bit disappointed with the first results of my installation.
Two days later I found the root cause - it was faulty 8Gb FC SFP. While troubleshooting I had to check everything - configuration, firmware, drivers, etc,  and that is when I found out that I had no idea neither how to check ESXi drivers' version nor how to update them. Quick search on the internet didn't give me a good step-by-step procedure, so I had to go through vSphere Command-Line Interface Installation and Scripting Guide and thought it would be good to document the entire procedure for the future use.

Basically, you have 2 options - you can either use vMA or you can install vCLI on Windows computer. With vMA you will need to first copy drivers to vMA. I am a lazy person, so I temporarily opened Remote Tech Support (SSH) on each host and used my workstation to update drivers.

The first thing you need to start with is to check is a current driver version.

Networking Driver

1. In vSwtich properties check the Network Adapter driver currently used . In my case it is bnx2x.
   
   
   
   
   
2. Login into ESXi via SSH with root account.
3. Run the following command - esxupdate query --v | grep bnx2x. The driver version starts right after 400. In the screenshot you can see there are 2 driver versions. These are the old driver that was deployed during the ESXi installation and the newer one which was installed with one of the ESXi  firmware upgrades.  

HBA Driver

1. Login into your ESXi host via SSH.
2. Change directory to /proc/scsi/qlaxxx (QLogic based) or /proc/scsi/lpfcxxx (Emulex based)
3. You will find several files (one file per each SCSI HBA adapter), in my system there are just two of them

4. Looking into the content of these files you can get driver version.

Now we are done with collection of the information. The next step is the driver upgrade proccess itself. The upgrade steps are the same for networking and HBA drivers. First you need to look for the latest verion of drivers you need at the VMware Download Center. 

1. Put the ESXi server into the maintenance mode - either from GUI, or from vCLI using command vicfg-hostops.pl --server SERVERNAME --operation enter

2. From vCLI run the following command. Make sure you have put the correct server name and path.

•  vihostupdate --server ESX01 --scan --bundle c:\update\hba.zip

This will tell you which of the drivers in your zip files are applicable and still not installed  on your ESXi host.

3. Once the output of previous command confirms everything is fine run the next command:

•  vihostupdate --server ESX01 --update --bundle c:\update\hba.zip

4. Repeat steps 2 and 3 for networking drivers.

5. Reboot the host and exit maintenance mode.

I would really appreciate any comments about quality of my posts (including my English :) ) and any wrong conclustion or mistakes I could make here.

If you find this post useful please share it with any of the buttons below.