Apple Fusion Drive

This post a bit off the theme of my blog, but I thought it might be useful for someone. I wrote it for my friends on facebook, who are mostly not technical persons, that's why I omitted technical details here. 

And yes, I am a big fan of Apple products. 

What I have found out talking with my colleagues recently is that even technical people are really confused about what the Apple Fusion Drive is.

Most of the people believe that Apple created another version of hybrid drive which has already existed on the market for a while, but to be honest it has never been a popular choice, though some geeks tout it constantly. 

Well, the truth is that Fusio

n Drive is a way bigger thing that a standard hybrid drive, which normally combines small amount of SSD storage (from 4Gb to 8Gb) and a traditional hard drive. Former is characterized by a comparably high price, but very fast seek time. That is, SSD spends about 0.1 ms to access the required bit of data. Latter, on the contrary, is significantly cheap, but quite slow. Its seek time is about 9-10 ms in a regular desktop or notebook hard disk.

SSD in hybrid disks is always used to cache recently written or read data only. Obviously, it speeds up hard disk`s performance, but in general the benefits are not good enough to cover all disadvantages, such as reduced lifetime, increased power usage, etc.

So, how is Fusion Drive better than Hybrid Drive? Let me start from the Enterprise solutions. That is where this technology came I believe. HP has been offering very interesting Multi-Tier Storage solution called 3Par. It is huge server rack (or several of those) equipped with hundreds and thousands of hard disks. This Storage hardware can work with different type of disks: SSD - the most expensive and the fastest one, FC - relatively fast and still expensive, FATA - cheap, slow and unreliable. Each type of disks serves its own purpose. SSD can be utilized for High Performance servers (for instance, SQL or Exchange) which can service thousands of requests per second. FC disks may be used for a bit less important production servers. FATA will be utilized as a storage for data which is rarely used or changed, for instance, your collection of movies. So, HP 3Par has intelligence to analyze how often the data is used and move it to the appropriate level. If some server is idle for a while its data will be moved to a slower storage level absolutely transparently for users, whereas another server, which currently experiences high load, will be moved to SSD. Generally speaking, HP 3Par is like a system administrator who works 24\7 and whose only responsibility is to put the data to the right type of disks to meet all performance requirements.

Phew, that was quite a long introduction. However, if you read through the whole text to this point I can simply tell you that Apple Fusion Drive is almost HP 3Par in terms of data management. Here are the main facts of the Fusion Drive:

1. SSD drive`s size is 128 Gb from which 4Gb are dedicated to cache write operations.
2. MacOS constantly tracks disk activity and analyze what files are used most of all. Using this data MacOS moves data to either SSD, if this data is used heavily ,or to the hard disk if the data is used a little or none.
3. Data is moved on the block level, not on the file level. Imagine, you have a huge video (50 Gb) recording of your last birthday, which is stored on a traditional hard drive. You want to edit it and remove some embarrassing moments of yourself sleeping in a salad bowl. You open it in iMovie and start to work with the last 5 minutes of the video. MacOS sees that the file is being edited and used heavily, but it won't copy the entire 50Gb file to SSD drive. It is smart enough to copy to SSD only those blocks of the video file you are actually working with.
4. The last experiments with this feature showed that it is fully programmed in software. This means you can easily implement and use this functionality in older generations of MacOS computers, either iMac or MacBook.
5. It is as fast as SSD and still provides huge amount of disk space (up to 3Tb) for users. If you were to buy 3Tb of SSD you would need to be ready to spend about 7,500 USD. Easy to compare, huh? :)

I know this stuff is quite boring for most of you, but for Apple fans or those who is currently contemplating about buying one of the new iMac of Mac Mini this information can be quite useful. :)

If you find this post useful please share it with any of the buttons below.

vCenter Alarms - bulk "Set Email Notification" change

Recently I have installed a new vCenter server and, as a good practice, was going to to configure the most critical Alarms to send email to my mailbox. Surely, it was going to be manual process as I have zero scripting knowledge except the PowerCLI minimum which was part of VCAP exam. But frankly speaking, I barely remember half of the PowerCLI command  I used to know.

However, I thought I might try to refresh my rusty PoweCLI knowledge. Therefore, I picked up the first article in search results which provided a good PS script for vCenter 5.0 and useful links for automation of the boring process of setting "Send Email" action for all Alarms. Definitely, there is still no descent explanation why VMware hasn't provided such configuration option.

While I was applying this script to my infrastructure I found out that number of Alarms differ from version to version. Here, in my company we have vCenter 5.0 Update 1a in production and vCenter 5.1 in test environment. Therefore, I have decided to adjust the script for both our versions.

So, here are the links to the scripts which might be useful for you.

• Set Email Notification script for vCenter 5.0
• Set Email Notification script for vCenter 5.0 Update 1a
• Set Email Notification script for vCenter 5.1
• Excel file with all Alarms categorized by vCenter version - god knows why you would need it :)

There are plenty opportunities to customize these scripts for more sophisticated actions, but for my scarce PoweCLI skills just an email is more than enough :)

If you find this post useful please share it with any of the buttons below.

Great collection of vSphere Best Practices

I really like reading Best Practices. They broaden one's horizons, but the best thing about them is that they give you plenty of ideas what to read about and provoke to go deep into technologies. You read them through and make list of things you want to try or would like to implement in your production environment. Then you familiarize yourself with new technologies reading Technical Resources documents on VMware site. You end up with myriad of tabs opened in your browser and you think you have to read all of them and even comment some of the posts. Well, that is how it works for me.

So, here is the best collection of the Best Practices - http://vmwaremine.com/vmware-vsphere-best-practices/

I have already started bookmarking the most interesting docs.

If you find this post useful please share it with any of the buttons below.

Fixing Site Recovery Manager credentials

Today I have discovered that my Site Recovery Manager is not working anymore. The error was obvious - the credentials which were used on SRM server to connect to vCenter had been changed.

Very soon I found VMware KB explaining how to update these credentials. I accurately followed the steps in KB, but I have gotten another problem.

Well, I used the command and set the credentials for the SRM Database, but while learning the syntax of installcreds command I discovered that there is an option to use installcreds for vCenter credentials. 

So I ran installcreds.exe -key "vcenter.example.com" -u domain.com\username, restarted SRM service and voila!, everything went back to normal.

Moreover, the original command from VMware KB hasn't worked for me at all, even after set credentials for SRM database, which made my findings even more useful :)

If you find this post useful please share it with any of the buttons below.

VMware Hands-on Labs from VMworld 2012 are being offered online!

Guys,

if you have missed VMworld 2012 you still have an opportunity to access VMware Hands-On Labs online which were offered during the VMworld 2012. The registration has been opened a couple of days ago. So you still have a good chance to be amongst the first luckiest persons to play with new VMware products without tiresome and time-consuming process of building home lab for each of the scenarios.

So, proceed with the Registration and be ready to provide a good feedback :)

Personally, I have spent already half a day to rebuild my home lab with vSphere 5.0 so as to be able to test and to document the upgrade steps for vSphere 5.1. However, if you have a look at the list of possible labs you will understand how many efforts and time you will save using pre-configured and well-instructed labs from VMware.

If you find this post useful please share it with any of the buttons below.

The new certification roadmap from VMware

Well, VMware is way more faster at generating news than I am at processing them. Still catching up with all interesting pieces of news I have missed while being on vacation.

In the shade of the bright and sparkling announcement of vSphere 5.1 the introduction of new Certification Roadmap has been overlooked by many bloggers.

Here is how it looks like now

The picture depicts four areas of certification

• Cloud - this is mostly to prove you are good at administering vCloud Director (vCenter Chargeback, vCloud Networking and vCloud Organizations)
• Datacenter Virtualization - this certification shows your expertise in administration of vSphere
• End User Computing - this is all about building Virtual Desktop Infrastructure
• Cloud Application Platform - purely developers' area.

Each category is divided into 2/3 levels showing different levels of skill sets. Some of the exams are not available yet, but coming out soon.

Currently I am trying to figure out my own certification roadmap for the next year and I still don't know which direction to go - Cloud or End User Computing. Both fields look very interesting, therefore first I need to understand which area has more prospects and potential demand on job market. Though first I have to site VCAP5-DCA which is fortunately now available to take in my country.


If you find this post useful please share it with any of the buttons below.

Licensing changes brought by vSphere 5.1 release - no vRam entitlements

While I have been torn apart by temptation to learn all new features of vSphere 5.1 by creating a long list of documents and blogs to read I have missed probably the most important change in VMware vSphere Licensing model - the is no more vRAM entitlement/limitation.

VMware claims they have carefully listened to their clients:

"Our customers spoke clearly," VMware Chief Marketing Officer Rick Jackson said at a press conference after Monday's opening keynote. "They don't want to think about things like vRAM. We're changing it back to what our customers want."

To me it seems more like another answer to Microsoft Hyper-V 3.0 which is about to be released and has already changed the virtualization market's landscape. But I personally liked the fact that VMware has enough courage to admit they had made a mistake with presenting vRAM.

"Yes, it is an admission that we had made things overly complex, and we are rectifying that mistake," outgoing VMware CEO Paul Maritz says bluntly. "Mea culpa."

So, there is no more vRAM and Cores per CPU limits now. Plain and clear licensing model with only requirement to have 1 license per each CPU/Socket in your server. It is quite an improvement which definitely makes life of VMware customers and VMware salesmen easier. I guess the total number man-hours spent last year worldwide in debates about vRAM could have been sufficient to build a space ship.

Another significant benefit of new licensing and pricing model is that all vSphere editions (except Essentials Kit) have gotten plenty of nice features included:  vSphere Replication, vShield Endpoint, High Availability and Data Protection (previously known as VMware Data Recovery). 

Free vSphere Hypervisor's limitations have caused some confusion as different part of VMware website and documents were providing different information. Here is the latest and most precise information from GabesVirtualWorld

“There is no vRAM in vSphere 5/5.1, including the free vSphere Hypervisor. If a host licensed with vSphere Hypervisor has more than 32GB of physical RAM it will error when applying the license or on boot. Before the vSphere Hypervisor license is applied, it will be running in 60-day evaluation mode (functionally equivalent to Ent+).  So in the case of a host with more than 32GB of physical RAM, when assigning the vSphere Hypervisor license fails the host will remain in the evaluation mode. If they do want to proceed they either need to find a box with 32 or less or go into the BIOS of the offending machine and turn off the additional RAM.”

The last interesting new fact is the special vSphere Standard edition which includes vCenter Operations Management Suite Advanced and vCenter Protect Standard. I am wondering why there is no similar Enterprise Edition? 

Now I am looking forward to the 11th of September when vSphere 5.1 will be available for download so I can start rebuilding my home lab.


If you find this post useful please share it with any of the buttons below.

VMware vSphere Replication

Just 1 week ago I have finished building Disaster Recovery solution using vSphere Site Recovery Manager product. At this time vSphere Replication was solely part of SRM, but not of the vSphere. That's why I was a bit surprised when I saw "Introduction to vSphere Replication" as a part of "What's New in vSphere 5.1" documentation. At first sight it looked as a competitive contradiction to a current Disaster Recovery solution - SRM. But when I started to read the documentation I suddenly remembered that when I was playing around with Windows 2012 Release Preview I noticed that Microsoft included new free feature called Hyper-V Replica, which was supposed to provide Disaster Recovery functionality. Considering that it was absolutely free of charge benefit it had at least one strong advantage over VMware SRM.

So now it has gotten clear that this is just an answer to Microsoft challenge on Virtualization Market. When I went further into What's New documentation I found some other obvious evidence that VMware is striking back on Microsoft Server 2012.

Now, after I have done some reading I can see that vSphere Replication can replace SRM in small companies only. Here are the reasons:

• Having it working in your infrastructure doesn't provide automation/orchestration of Disaster Recovery scenario. The same statement applies to Hyper-V Replica. You will need either powershell scripts or System Center Orchestrator to get automation options. 
• Each Virtual Machines at the Recovery Site has to be powered on manually
• You will need to reconnect each your VM to the correct network.
• There might be necessity to Renumber IP addresses of some VMs - again, manually.
• You will miss possibility to test your recovery plan during working hours since you cannot power on Replica VM if the original VM is still running and reachable.
• There is no failback option in vSphere Replication. 

Well, it must be admitted that if you have all your steps of Disaster Recovery scenario well-documented and throughly tested you may deal with it just fine with probably 10 to 100 virtual machines. 

However, every single manual action significantly increases risk of human error, especially when every minute matters. Multiply all steps need to be taken by hundred/thousand VMs running in big enterprises  and you understand why it is not applicable. You can also take into the account the fact that there is not always a qualified person at the Recovery Site who can take care of all DR procedures properly.

Nevertheless, it looks like an adequate answer to Microsoft, considering that you can protect all your VMs at no cost if you own license starting from Essentials Plus Kit to Enterprise Plus Edition. An ideal solution for small companies seeking how to cut expenses on Disaster Recovery.

If you find this post useful please share it with any of the buttons below.

VMware vSphere 5.1 Clustering Deepdive is available now

Duncan Epping and Frank Denneman, two very probably the best VMware evangelists, have just released the new book covering every single bit of vSphere 5.1 Clustering.

I have bought and read both previous books of them (both in paper and in kindle format, for the simple reason that I can afford it :) and I can say that these books have significantly contributed into building my knowledge and clear understanding of HA and DRS cluster's mechanic and logic. I personally believe that HA/DRS functionality is a heart of all vSphere package and therefore the solid expertise of vSphere cluster is a must for anyone who sees his future in virtualization world.

So, choose your preferable format and enjoy reading.

• Paper copy – $ 24.95
• Kindle version – $ 7.49

vSphere 5.1 is released

Update:  The vSphere 5.1 and vCloud packages are available for downloading as of today.  Enjoy. I will have couple of sleepless nights for sure :) 

Being on vacation I have missed the release of my favourite product in IT world. I am trying to catch up now reading all "What's new" documents, but I definitely miss my lab to have some hands-on experience with all new fantastic features of vSphere 5.1.

Here are the links for all of the documents that can give you a brief overview of most interesting innovations in vSphere 5.1

• What's new in vSphere 5.1
• What's New in VMware vCenter Server 5.1
• What's New in VMware vSphere 5.1 - Platform
• What's New in VMware vSphere 5.1 - Networking
• What's New in VMware vSphere 5.1 - Storage
• Technical White Paper: What's New in VMware vSphere 5.1 - Performance
• Technical White Paper: Introduction to vSphere Replication
• Technical White Paper: Introduction to vSphere Date Protection

There are probably some other VMware products which have had new versions, but I think the links above cover most of the important topics.

Reclaiming Disk Space from Thin Disks

There are huge number of posts describing how to reclaim disk space and to decrease used storage for Thin Disks. However, very few pay attention or emphasize one precaution measure you have to take before proceeding.

Here is the whole story. There are situations when you temporarily COPY large amount of data on Thin Provisioned Disk. This action inflates disk's VMDK file on VMFS datastore. When you delete this data off the Thin Disk  the VMDK file size stays the same and that is because the data is not deleted, but simple marked to be overwritten. So you end up with situation when actual disk size in Virtual Machine is, let's say, 10 GB, but its VMDK occupies hundreds of GBs on VMFS Datastore.

Basically, this problem is resolved in two steps.

1. You run SDELETE utility (link) on your drive to zeroize all unused disk space. This way you will let vSphere distinguish between actually used blocks of disks and unused disk space.
2. Once it is over you need to start Storage vMotion of your Thin Disk. The main point here is that you have to move your Disk/VM to a VMFS Datastore with different Block Size. Under this condition only vSphere will invoke legacy datamover which doesn't copy zeroized data blocks. As usually, Duncan has the best explanation of the block size impact.

With VMFS-5 which has unified Block Size of 1 Mb this is not as easy as pie. First you will need to create a new VMFS-3 datastore with different block size and then it has to be upgraded to VMFS-5.

And here comes the  !Warning!

• Pay Attention to the amount of Free Space on your VMFS Datastore before running SDELETE.

As I have said  SDELETE will zeroize all free space on you Thin Provisioned Disk. That means it will inflate VMDK file to its maximum provisioned space. So you have to make sure that SDELETE will not gobble all your free space on a VMFS datastore.

If you find this post useful please share it with any of the buttons below. 

I am finally back

Well, I know for sure I am not getting vExpert status next year. Have been utterly busy with preparation for IELTS exam over the last 5 months. Just yesterday I have got the results I sought (Overall: 8.0).

I have been working with myriad of technologies this year, except VMware products :)  Therefore, I have got a bit rusty in a virtualization world and need a deep dive for at least a month into virtual products before I can get back to the level I was last year.

This week I am planning to write about HP Virtual Library System and not very positive experience with this product, though it could be just a result of using VLS together with Veritas Backup Exec.

Later on, I am going to share my experience with VMware Site Recovery Manager which I have planned and configured for our company last week.

And the last great piece of news is that now we have an authorized examination center in Kazakhstan where we can take VCAP 5 exams. I remember that even in Italy there was no such place where I could sit VCAP 4 exam last year and had to go to Switzerland.

See you soon guys.

PS I was really impressed by the fact that average number of visitors haven't decreased lately even though I haven't posted for so long. Thanks to you all for reading my blog.

Shame on me

Well, this is something really embarrassing.

I have upgraded our vSphere to version 5 about 3 months ago and it is only NOW I found out that I haven't upgraded Distributed vSwitch. To be honest, I even didn't know I had to do it manually, I was convinced it would be upgraded during the ESXi hosts' upgrade and very probably I overlooked this fact in the official documentation.

It doesn't impact your infrastructure at all, however your Distributed vSwitch will lack cool new features like NetFlow, Port Mirroring and User-Defined network resource pools.

So, if you missed this part too just go to dvSwitch, where you will notice Yellow warning mark next to Upgrade link. The upgrade process will check if your hosts are compatible with dvSwitch 5.0. After another couple of clicks you will have it upgraded.

This just proves I am getting too rusty on vSphere due to absence of practice with it. I think I need to change my job :)

My first vExpert 2012

It has been 8 days since I received "Welcome on vExpert board" mail from VMware Community Manager, and I tempted to make a note about such big event for me here, but you can't imagine how complicated and time-consuming a preparation for IELTS exam can be  :)

Well, to be honest, I was 99% sure I would get it considering how much time I have devoted to my new virtualization passion, especially taking into account the fact I was blogging in 2 languages. I know that quality of my posts still leaves much to be desired, but I really hope I am doing better and better with each new post.

I said I was expecting it, but it doesn't mean I wasn't really glad and proud of myself for getting vExpert 2012 title. Unfortunately, the new job I got doesn't give me a lot of chances to work with vSphere, therefore, I have had not much to post about this year. However, with vExpert status came the motivation to study VMware products and write more about them. Otherwise, I won't get my second vExpert next year. :)

Quite soon I will be involved into Disaster Recovery site planning and implementation phases, so I am looking forward to get my hands on it. Then I can spend more time with my blog and its readers.

Talk to you soon.

Wanna move or copy virtual machine over WAN?

What do you usually do to copy or move over WAN link from one site to another? How do you do it if the remote site is 5000 miles away and you have only 2Mb E1 link between sites?

Normally, I would copy VM from a datastore to, for instance, local disk on vCenter server. Then I would copy VM files to a server at a remote site and once copy process is over I will copy the files from the server at the remote site to the datastore at the remote site. Not the most elegant and fastest method, but it worked for me.

However, yesterday we faced a situation when we needed to copy VM over 2Mbit link. VM had thin disk with 120 GB provisioned disk space and 7 GB of actual allocated disk space. Obviously, regular way of moving files wouldn't work here for very simple reason - once you copy VM with thin disks to NTFS partition those disks are converted to thick type. Thus, I would have to copy 120 GB over 2 Mb link which in ideal scenario could take me about 7 days.

Sure, the very first thought I had was to use VMware Converter to changed provisioned disk space to the actual one. However, it wasn't the most convenient way. Failed twice for me because I overlooked couple of the settings in wizard. That's when my colleague advised me to try OVA/OVF

.

Well, for me OVA/OVF was always a standard of multiple virtual appliances I import into vSphere, but I never thought about this format as of a way of exporting and moving VM around (Yes, I know, I am narrow minded person).

Once I tested OVF import/export operations I found 4 cool advantages:

1. It is really really simple. Shut down VM, go to File - Export - Export OVF Template, choose between OVA or OVF and where to save the VM. That's it.

2. It does save only actual data. So, you don't have have a problem of disks being converted to thick once they are copied to NTFS.

3. It compresses VM very well.

If you use GUI wizard you can't select compression rate, but in my situation I got 1.3 Gb OVA file out of VM with 7 Gb allocated disk space. You can download and install VMware OVF tool which will provide you with command line for OVA/OVF operations and way more of options, including different compression rates.

Just to check how good the compression rate is I exported couple more of VMs and both of them were compressed to have of the original size of actual used storage. 

4. Not sure if it works for all filesystems and OS, but when I exported my Linux VM to OVA file and then imported it back it shrinked the thin disk to the actual size. So, this might be useful, for instance, for situations when you copied big amount of data to VM with thin disks and afterwards you delete this data leaving thin  disk inflated but not efficiently used.

N. B.  Be careful with VMs running some kind of licensing services which are tightly bound to VM's mac address or some other type of ID of virtual hardware because when you import OVF you get VM with new virtual hardware, so virtual machine OS has to adjust itself.

If you find this post useful please share it with any of the buttons below. 

Upgrade to vSphere 5 - Brief checklist

Recently I have completed the upgrade of one small vSphere 4.1 farm to vSphere 5. Quite soon I will need to upgrade another vSphere 4.1 farm with similar specs, so I decided to write down the brief checklist.
This is not a universal upgrade scenario and it is very specific to our hardware and vSphere configuration, which I will provide below, however, this can be a good high level plan which you can adjust to your environment.


Our vSphere enviroment is hosted on 4 HP Proliant BL460c G7 servers in HP BladeSystem C7000 Enclosure. We use HP FlexFabric VirtualConnect modules to connect to SAN and Ethernet switches. The storage space is provided by EVA 6400.


All ESXi hosts had the latest ESXi 4.1 U2 firmware. vCenter is installed on a physical server and was connected to another vCenter server by Linked Mode. There are about 80 VMs, mostly Windows 2003 and 2008 R2.


So, here is very brief plan for vSphere 5 upgrade, no screenshots or monkey-donkey procedures are provided :) 


0. Build Test Lab


If you have spare hardware with similar specs build a test lab to reproduce the current infrastructure. Test and document the upgrade process. 


I spent a week to build identical vSphere farm including couple of virtual MS Failover clusters. This gave me confidence that I can easily failover MS cluster services between nodes running on different ESXi versions.


1. Compatibility List


This was the most difficult and time consuming part of the upgrade. You need to start with VMware Compatibility Guide. It is very extensive. Make sure you have tested every single part of your infrastructure against this HCL, e.g. network driver, HBA firmware, server's BIOS, SAN switch firmware. That will help you to stay on the safe side and be able to get support from VMware in case you face issues with vSphere 5.If you have similar hardware you can find this post useful.


The VMware HCL is not the only place to check. In my case I had to check HP Single Point of Connectivity Knowledge (SPOCK) to verify approved firmware/driver versions for vSphere from the HP standpoint.


You will for sure find some components to be updated and probably you will need to schedule some maintenance time during the weekend. In my case I had to upgrade HP EVA 6400 Firmware to the latest version to meet vSphere 5 prerequisites. 


Once you align your infrastructure with Compatibility List you can proceed with step 2. 


2. Host Agent Pre-Upgrade Checker


Run Host agent pre-upgrade check to make sure the new vCenter agent will be installed sucessfully. 


This is quite simple task that will check if VPX agent can be upgraded successfully. Make sure you run it as administrator otherwise it won't show your system DSN for vCenter database. 

3. Upgrade vCenter

a. Uninstall Linked Mode


vCenter upgrade guide says it will uninstall Linked Mode automatically as versions 4 and 5 are not compatible with regard to Linked Mode. However, I prefer to keep it in a controlled way.

b. Check Database Compatability Level and permissions

You will need to check the following document for specific details on your database type

c. Check vCenter prerequisites

The same document will be useful in this case again.

d. Backup vCenter database. 


Not much to tell about it :)


e. Run vCenter Installation Wizard


All you will need here is clicking Next and credentials for vCenter Database.

4. Upgrade vSphere Update Manager


In my situation VUM was not really useful. First, I planned to use for the hosts' upgrade, but it failed twice. Therefore, I left this as a last task.

5. Upgrade Hosts

a. Put the host in Maintenance mode.


b. Disconnect all LUNs 


You can unplug FC cable, unpresent LUNs, disable ports on SAN switch. With HP Virtual Connect I simply unassigned FC SAN from server's profile.

c. Be careful by choosing the right ESXi image for upgrade


If you installed previous version of ESXi with HP customized image be sure to use HP ESXi 5.0 image for upgrade. You can download it here. 
I didn't pay attention to this fact so my first attempt of upgrade with VUM failed - newly upgraded host couldn't be added to vCenter. This particular issue is explained in HP Support Document.

d. Consider upgrading directly from ESXi ISO


Second attempt of host's upgrade with VUM failed as well. The installation process stuck at “About to install” stage. Seems like HP ESXi  5 ISO doesn't support upgrade with  VUM. If you don't have big number of hosts it is not a big deal.

e. Reconnect Host


Once the host is upgraded you need to add it to vCenter and get SAN connectivity back.

d. MS Failover Cluster pRDM disks


If you have virtual MS Failover Cluster nodes in you vSphere you will need to run the following command in order to fix the long boot time issue.
esxcli storage core device setconfig -d device .naa --perennially-reserved=true for each LUNs presented to virtual

Here you can find additional details.

6. Upgrade/Migrate to VMFS-5


You very probably read about VMFS-5 and difference between new and upgrade VMFS-5 datastores. For me the biggest driving force to go with new VMFS-5 datastores was the fact that block size was different across old datastores. That was causing slow Storage vMotion. Duncan Epping gave very nice explanation of the nature of this problem. 
If you don't spend days and weeks (depends on the number of your datastores and your storage performance) moving your VMs around you'd better have very clear understanding about your final Datastore design. Considering new features like Storage DRS and Profile-Driven Storage this can take you quite some time.

7. Updating VMware tools.


Depending on how critical your VMs and their services are you can go either with VUM VMware tools upgrade or manual upgrade.

8. Upgrade VM hardware


The same story here. We don't have hundreds and thousands of VMs, therefore, I preferred to upgrade virtual hardware manually. This gives me  feeling of control :) if your VMs have Hardware Version 7 you can take advantage of VM snapshots to be able to revert Virtual Hardware upgrade. 




Sure, this checklist is not even close to the proper upgrade procedure, but this might be a good starting point for your own upgrade plan.




Feel free to give me your feedback. Really appreciate it.


Update 1


Do not forget to upgrade youк dvSwitch to version 5 as I did. It has to be done once your vCenter and ESXi hosts are upgraded.


If you find this post useful please share it with any of the buttons below. 

Slow boot of ESXi running virtual MS Failover Сluster is finally solved in vSphere 5

We have quite a few virtual MS Failover Clusters in our vSphere farm. Usually these are File&Print and SQL clusters. Some people say MS clustering is old fashioned way of providing HA for most important users' services.  Having them in virtual environment makes vSphere maintenance possible only during non-working hours for the simple reason that you can't vmotion cluster nodes off the ESXi hosts while they are powered on. Another big is a slow boot time of ESXi hosts that have presented shared RDM disks for MS Failover cluster. During boot process ESXi host tries to claim all disks it can see, but since shared cluster RDM disks have Persistent SCSI reservation placed by active MS Cluster node, such claim request will eventually time out. If you have a lot of such disks you can easily send your ESXi host for reboot and take a 40 mins nap. Maintenance of vSphere farm with 10-20 host turns out into a boring weekend.

VMware acknowledged this problem quite a long time ago, but all suggested workarounds were not efficient. The boot time was still slow.  

Yesterday, I finally tested upgrade to vSphere 5 and documented all the steps I will need to take for the upgrade of production environment.The first thing after upgrade I noticed was even longer boot time that quite disappointed me. I was inclined to think there would be built-in improvement in vSphere 5 for such a long lasting problem, but I was wrong. So, again I quickly googled and found the old VMware KB about this issue, but now it was updated with vSphere 5 information.

Now there is absolutely different solution for ESXi 5. With ESXi 4.1 you have to decrease Reservation Conflict Value Timeout (Scsi.CRTimeoutDuringBoot). With ESX 5.0 you manually tell ESXi hosts which LUNs are constantly reserved. Thus, ESXi will not even try to claim these disks. 

VMware used a word perennially, which I have never heard before :)

You will need to SSH into each ESXi host and run the following command 

esxcli storage core device setconfig -d <naa.id> --perennially-reserved=true 

where naa.id is RDM disk id that is used by MS Failover cluster.

Once I have everything properly configured it took my ESXi server less than 2 minutes to boot. Amazing improvement.

Here is the VMware KB with full information about this issue - KB Article 1016106.

PS  I am still looking for alternative solutions to fully get rid of virtual MS Failover clusters :)

If you find this post useful please share it with any of the buttons below. 

My VCAP-DCA experience

Well, I am back on the rails after two and half months of silence. So many things have happened during this period - the preparations for the exam, relocation back to my home country with whole my family and pets, change of the employer. But let me tell this story from the beginning, maybe it is not going to be the most interesting reading, but for me it is one of the "My Dear Diary" moments. You can always skip the story and go directly to practical notes about the exam and the preparations.

I decided to start my VCAP-DCA study in April, 2011. Even though, I am busy with so many different vendors and technologies in my daily work routine I decided to focus on virtualization and specifically on VMware products. For the beginners in VMware technologies like I was the official blueprint was just too much and scaring to start with and I really needed some advice on how to plan my preparations. The first and one of the best VCAP-DCA guide I found was Sean Crookston's web site. I started very slowly with it. Even though the exam is all about your practical skills with vSphere there are of hundreds of articles to read and dozens of training videos to watch. I could spend a week reading different blogs and vmware documents about Transparent Page Sharing, another week I could read only about NUMA technologies. Then summer came and melted my ambitions and motivation. In July I got seriously distracted by vSphere 5 release. August  traditionally was devoted to a vacation. In September I decided to take advantage of my VCP4 certificate and sit the VCP5 exam without taking VCP5 training.  It was my personal experiment on taking exam without any real life experience with vSphere 5. Surprisingly, it went quite well.

At the end of October I was finally notified on my last day of staying in Italy (which was 22nd of December) and only then I clearly realized that from my home country I would need to travel at least to Moscow just to take this exam. Assuming that I considered my chances to pass this exam lower than 50% I doubled my expected expenses for the exam. The final calculation showed that it might cost me about 2000 USD.  Therefore, I decided to take it in Europe, though I was very short on time.
At that moment  I had covered only 3 sections out of 9 from official blueprint. That's when I found another great VCAP-DCA guide from Edward Grigson. This guy has done stunning job. I think his guide is the fullest and the most comprehensive guide on the Internet right now. I reckon Edward should start selling it.

The bad thing about my decision to take the exam on 20th of December was overlapping schedule with my wife's ACCA exams. We have 2 children and that fact doesn't leave us, parents, much free time. Now we had to split that sparse free time between us. Moreover, we had to move to another country and I had to change a job which was making our situation even worse.We had to sell things, pack them, through them away, prepare documents for pets, find the proper school for my elder son, look for apartment, find new job, make a handover of all my working responsibilities to my colleagues. Guys, believe me that was a stressful mess.

The closest test center was in Zurich, which was about 3 hours of driving from Milan. The day before exam I checked weather forecast for Zurich, it was +9C which I considered safe to drive. I left home 6 hours before exam, early in the morning. In 1.5 hours, when I was somewhere in Alp mountains it started to snow. That was one of the hardest snowfall I ever saw in Italy. Well, weather forecast was totally wrong and I was on the fully snowed road in the car with summer tyres. I almost crashed couple of times and had to switch to first gear and drive at speed 20km/h for about 1,5 hour. Can't express how nervous I was considering that I had to sell the car the very next day and wasn't sure if could get to the test center on time.
Ok, I managed to get to the Zurich 1 hour before the exam. To make the things worse the fuse for cigarette lighter died so I knew my TomTom won't last more than 2 hours and getting home will be challenging. My head was full of bad thoughts about how the hell I am going to get back home.
Well, it is not the end of the story. I decided to take a quick lunch before exam. Ordered a salad with some poison-coloured sauce. When I was trying to get my wallet out I accidentally overturned the salad on my pants. At that moment I thought all these were just signs which were trying to say me I shouldn't have taken this exam.
When I showed up at the exam center I looked like some homeless guy - tired, nervous, with bright stains on my pants. Do you think it is the end of the story?  No way. When I sat at the computer and the lab started I discovered that keyboard had AZERTY layout. I thought I would be able to manage this problem, but first 15 minutes I was fighting my keyboard, particularly trying to find special characters which I needed to use in command line. I couldn't even type the password for vMA or Terminal session, instead I had to copy-paste it. Definitely, at this moment I was totally desperate and was already sure for 100% I would fail the exam. Luckily, the testing center administrator found another keyboard with QWERTY layout so we could swap keyboards and I could continue my lab in more familiar way. After 3,5 hours I still had 7 tasks uncompleted, but I was so exhausted and didn't have any will to fight. However, the rest of questions I fully answered and configured, though I wasn't sure I have done them correctly. So I had like 50/50 feeling about the results when I left.
I took exam on 20th of December, the results arrived on 13th of January. Not really 10 working days as VMware promises. My score was 350, but if it were even 301 I would be happy anyway. I didn't feel like I had strength to start it all over.

Practical notes:

1. VCAP-DCA guide from Edward Grigson contains almost everything you need to study about the lab. Just check what the difference is with the latest blueprint
2. I had to practice about 5-6 hours a day for about 7 weeks, non-stop.  
3. Very often preparing your lab to test some particular feature can take you 5 times longer than practicing that particular feature. In the beginning it was very annoying, but later on I recognized it helped me significantly on getting a lot of commands and actions on the tips of my fingers. 
4. If you have Linux command line experience you have serious advantage over the vSphere admins with Windows experience. I spent a lot of time studying basics of Linux.
5. I started preparations with virtual lab built on my powerful desktop at home, but then I got couple of spare HP blades at the office and also old EVA 3000. Honestly speaking, working with real hardware is a big advantage - it is faster and it is closer to real life (networking part for instance). At the end I used virtual lab very rarely. 
6. You have to practice every single feature mentioned in the blueprint. Although, in my lab I faced at least one thing that was not mentioned in the blueprint. Just read each section and question yourself if you can really configure it without looking in any VMware guide. Yes, you will be provided with vSphere documentation during the lab, but for the sake of saving time, imagine you won't have them at all. 
7. Leave the last week before exam to go briefly through all blueprint again.
8. Studying all blueprint doesn't mean you will be asked everything. You will be a bit disappointed that you are not asked to configure, for instance, vCenter Hearbeat even though you spent hours and hours studying the product you will probably never use. It is just matter of luck.
9. Memorize most popular Advanced Parameters of vCenter and virtual machines. I used some of them in real life, but never bothered to remember them. Most of the time I googled and then used copy-paste. When I had to find one during the lab I just wasted 5-7 mins without any luck.
10. The terminal access to the Lab is not fast. Switching between different windows takes time and sometimes is annoying. 
11. Some questions are not well worded. I had to re-read them several times before I could understand what they wanted me to do, but probably it is just me and my far from perfect English. 
12. Do easy questions first. If you see this question will take you some time to think about, make a note on your paper and go to the next one. 
13. The Biggest Surprise - some of the questions are insultingly easy. I mean, you spend days and nights to go deep into the vSphere technologies and then you are asked to configure something so simple that you think you misunderstood the question and there was some trick in it.
14. The Greatest Advice I read in one of the blog (sorry, can find the URL now) - write down questions' numbers in a row on the paper you will be given. This way you can track your progress by crossing out the questions you have completed and putting notes to questions you postponed. I am 100% sure if I hadn't read this question I wouldn't score the passing score.

It took me about 1 year and 5 month from the moment I first saw vSphere to the VCAP DCA certificate. If you don't have children yet, you are single, and full of passion for virtualization it shouldn't take you longer than a year :) 

Hope my story and notes can increase your chances or give you some guidance and probably motivation to pass this exam.

PS For those who are interested to know total number of people holding VCAP DCA certification - my ID is 553. At least I am in the first thousand. :)

If you find this post useful please share it with any of the buttons below.