Wanna move or copy virtual machine over WAN?

What do you usually do to copy or move over WAN link from one site to another? How do you do it if the remote site is 5000 miles away and you have only 2Mb E1 link between sites?

Normally, I would copy VM from a datastore to, for instance, local disk on vCenter server. Then I would copy VM files to a server at a remote site and once copy process is over I will copy the files from the server at the remote site to the datastore at the remote site. Not the most elegant and fastest method, but it worked for me.

However, yesterday we faced a situation when we needed to copy VM over 2Mbit link. VM had thin disk with 120 GB provisioned disk space and 7 GB of actual allocated disk space. Obviously, regular way of moving files wouldn't work here for very simple reason - once you copy VM with thin disks to NTFS partition those disks are converted to thick type. Thus, I would have to copy 120 GB over 2 Mb link which in ideal scenario could take me about 7 days.

Sure, the very first thought I had was to use VMware Converter to changed provisioned disk space to the actual one. However, it wasn't the most convenient way. Failed twice for me because I overlooked couple of the settings in wizard. That's when my colleague advised me to try OVA/OVF

.

Well, for me OVA/OVF was always a standard of multiple virtual appliances I import into vSphere, but I never thought about this format as of a way of exporting and moving VM around (Yes, I know, I am narrow minded person).

Once I tested OVF import/export operations I found 4 cool advantages:

1. It is really really simple. Shut down VM, go to File - Export - Export OVF Template, choose between OVA or OVF and where to save the VM. That's it.

2. It does save only actual data. So, you don't have have a problem of disks being converted to thick once they are copied to NTFS.

3. It compresses VM very well.

If you use GUI wizard you can't select compression rate, but in my situation I got 1.3 Gb OVA file out of VM with 7 Gb allocated disk space. You can download and install VMware OVF tool which will provide you with command line for OVA/OVF operations and way more of options, including different compression rates.

Just to check how good the compression rate is I exported couple more of VMs and both of them were compressed to have of the original size of actual used storage. 

4. Not sure if it works for all filesystems and OS, but when I exported my Linux VM to OVA file and then imported it back it shrinked the thin disk to the actual size. So, this might be useful, for instance, for situations when you copied big amount of data to VM with thin disks and afterwards you delete this data leaving thin  disk inflated but not efficiently used.

N. B.  Be careful with VMs running some kind of licensing services which are tightly bound to VM's mac address or some other type of ID of virtual hardware because when you import OVF you get VM with new virtual hardware, so virtual machine OS has to adjust itself.

If you find this post useful please share it with any of the buttons below. 

Upgrade to vSphere 5 - Brief checklist

Recently I have completed the upgrade of one small vSphere 4.1 farm to vSphere 5. Quite soon I will need to upgrade another vSphere 4.1 farm with similar specs, so I decided to write down the brief checklist.
This is not a universal upgrade scenario and it is very specific to our hardware and vSphere configuration, which I will provide below, however, this can be a good high level plan which you can adjust to your environment.


Our vSphere enviroment is hosted on 4 HP Proliant BL460c G7 servers in HP BladeSystem C7000 Enclosure. We use HP FlexFabric VirtualConnect modules to connect to SAN and Ethernet switches. The storage space is provided by EVA 6400.


All ESXi hosts had the latest ESXi 4.1 U2 firmware. vCenter is installed on a physical server and was connected to another vCenter server by Linked Mode. There are about 80 VMs, mostly Windows 2003 and 2008 R2.


So, here is very brief plan for vSphere 5 upgrade, no screenshots or monkey-donkey procedures are provided :) 


0. Build Test Lab


If you have spare hardware with similar specs build a test lab to reproduce the current infrastructure. Test and document the upgrade process. 


I spent a week to build identical vSphere farm including couple of virtual MS Failover clusters. This gave me confidence that I can easily failover MS cluster services between nodes running on different ESXi versions.


1. Compatibility List


This was the most difficult and time consuming part of the upgrade. You need to start with VMware Compatibility Guide. It is very extensive. Make sure you have tested every single part of your infrastructure against this HCL, e.g. network driver, HBA firmware, server's BIOS, SAN switch firmware. That will help you to stay on the safe side and be able to get support from VMware in case you face issues with vSphere 5.If you have similar hardware you can find this post useful.


The VMware HCL is not the only place to check. In my case I had to check HP Single Point of Connectivity Knowledge (SPOCK) to verify approved firmware/driver versions for vSphere from the HP standpoint.


You will for sure find some components to be updated and probably you will need to schedule some maintenance time during the weekend. In my case I had to upgrade HP EVA 6400 Firmware to the latest version to meet vSphere 5 prerequisites. 


Once you align your infrastructure with Compatibility List you can proceed with step 2. 


2. Host Agent Pre-Upgrade Checker


Run Host agent pre-upgrade check to make sure the new vCenter agent will be installed sucessfully. 


This is quite simple task that will check if VPX agent can be upgraded successfully. Make sure you run it as administrator otherwise it won't show your system DSN for vCenter database. 

3. Upgrade vCenter

a. Uninstall Linked Mode


vCenter upgrade guide says it will uninstall Linked Mode automatically as versions 4 and 5 are not compatible with regard to Linked Mode. However, I prefer to keep it in a controlled way.

b. Check Database Compatability Level and permissions

You will need to check the following document for specific details on your database type

c. Check vCenter prerequisites

The same document will be useful in this case again.

d. Backup vCenter database. 


Not much to tell about it :)


e. Run vCenter Installation Wizard


All you will need here is clicking Next and credentials for vCenter Database.

4. Upgrade vSphere Update Manager


In my situation VUM was not really useful. First, I planned to use for the hosts' upgrade, but it failed twice. Therefore, I left this as a last task.

5. Upgrade Hosts

a. Put the host in Maintenance mode.


b. Disconnect all LUNs 


You can unplug FC cable, unpresent LUNs, disable ports on SAN switch. With HP Virtual Connect I simply unassigned FC SAN from server's profile.

c. Be careful by choosing the right ESXi image for upgrade


If you installed previous version of ESXi with HP customized image be sure to use HP ESXi 5.0 image for upgrade. You can download it here. 
I didn't pay attention to this fact so my first attempt of upgrade with VUM failed - newly upgraded host couldn't be added to vCenter. This particular issue is explained in HP Support Document.

d. Consider upgrading directly from ESXi ISO


Second attempt of host's upgrade with VUM failed as well. The installation process stuck at “About to install” stage. Seems like HP ESXi  5 ISO doesn't support upgrade with  VUM. If you don't have big number of hosts it is not a big deal.

e. Reconnect Host


Once the host is upgraded you need to add it to vCenter and get SAN connectivity back.

d. MS Failover Cluster pRDM disks


If you have virtual MS Failover Cluster nodes in you vSphere you will need to run the following command in order to fix the long boot time issue.
esxcli storage core device setconfig -d device .naa --perennially-reserved=true for each LUNs presented to virtual

Here you can find additional details.

6. Upgrade/Migrate to VMFS-5


You very probably read about VMFS-5 and difference between new and upgrade VMFS-5 datastores. For me the biggest driving force to go with new VMFS-5 datastores was the fact that block size was different across old datastores. That was causing slow Storage vMotion. Duncan Epping gave very nice explanation of the nature of this problem. 
If you don't spend days and weeks (depends on the number of your datastores and your storage performance) moving your VMs around you'd better have very clear understanding about your final Datastore design. Considering new features like Storage DRS and Profile-Driven Storage this can take you quite some time.

7. Updating VMware tools.


Depending on how critical your VMs and their services are you can go either with VUM VMware tools upgrade or manual upgrade.

8. Upgrade VM hardware


The same story here. We don't have hundreds and thousands of VMs, therefore, I preferred to upgrade virtual hardware manually. This gives me  feeling of control :) if your VMs have Hardware Version 7 you can take advantage of VM snapshots to be able to revert Virtual Hardware upgrade. 




Sure, this checklist is not even close to the proper upgrade procedure, but this might be a good starting point for your own upgrade plan.




Feel free to give me your feedback. Really appreciate it.


Update 1


Do not forget to upgrade youк dvSwitch to version 5 as I did. It has to be done once your vCenter and ESXi hosts are upgraded.


If you find this post useful please share it with any of the buttons below. 

Slow boot of ESXi running virtual MS Failover Сluster is finally solved in vSphere 5

We have quite a few virtual MS Failover Clusters in our vSphere farm. Usually these are File&Print and SQL clusters. Some people say MS clustering is old fashioned way of providing HA for most important users' services.  Having them in virtual environment makes vSphere maintenance possible only during non-working hours for the simple reason that you can't vmotion cluster nodes off the ESXi hosts while they are powered on. Another big is a slow boot time of ESXi hosts that have presented shared RDM disks for MS Failover cluster. During boot process ESXi host tries to claim all disks it can see, but since shared cluster RDM disks have Persistent SCSI reservation placed by active MS Cluster node, such claim request will eventually time out. If you have a lot of such disks you can easily send your ESXi host for reboot and take a 40 mins nap. Maintenance of vSphere farm with 10-20 host turns out into a boring weekend.

VMware acknowledged this problem quite a long time ago, but all suggested workarounds were not efficient. The boot time was still slow.  

Yesterday, I finally tested upgrade to vSphere 5 and documented all the steps I will need to take for the upgrade of production environment.The first thing after upgrade I noticed was even longer boot time that quite disappointed me. I was inclined to think there would be built-in improvement in vSphere 5 for such a long lasting problem, but I was wrong. So, again I quickly googled and found the old VMware KB about this issue, but now it was updated with vSphere 5 information.

Now there is absolutely different solution for ESXi 5. With ESXi 4.1 you have to decrease Reservation Conflict Value Timeout (Scsi.CRTimeoutDuringBoot). With ESX 5.0 you manually tell ESXi hosts which LUNs are constantly reserved. Thus, ESXi will not even try to claim these disks. 

VMware used a word perennially, which I have never heard before :)

You will need to SSH into each ESXi host and run the following command 

esxcli storage core device setconfig -d <naa.id> --perennially-reserved=true 

where naa.id is RDM disk id that is used by MS Failover cluster.

Once I have everything properly configured it took my ESXi server less than 2 minutes to boot. Amazing improvement.

Here is the VMware KB with full information about this issue - KB Article 1016106.

PS  I am still looking for alternative solutions to fully get rid of virtual MS Failover clusters :)

If you find this post useful please share it with any of the buttons below. 

My VCAP-DCA experience

Well, I am back on the rails after two and half months of silence. So many things have happened during this period - the preparations for the exam, relocation back to my home country with whole my family and pets, change of the employer. But let me tell this story from the beginning, maybe it is not going to be the most interesting reading, but for me it is one of the "My Dear Diary" moments. You can always skip the story and go directly to practical notes about the exam and the preparations.

I decided to start my VCAP-DCA study in April, 2011. Even though, I am busy with so many different vendors and technologies in my daily work routine I decided to focus on virtualization and specifically on VMware products. For the beginners in VMware technologies like I was the official blueprint was just too much and scaring to start with and I really needed some advice on how to plan my preparations. The first and one of the best VCAP-DCA guide I found was Sean Crookston's web site. I started very slowly with it. Even though the exam is all about your practical skills with vSphere there are of hundreds of articles to read and dozens of training videos to watch. I could spend a week reading different blogs and vmware documents about Transparent Page Sharing, another week I could read only about NUMA technologies. Then summer came and melted my ambitions and motivation. In July I got seriously distracted by vSphere 5 release. August  traditionally was devoted to a vacation. In September I decided to take advantage of my VCP4 certificate and sit the VCP5 exam without taking VCP5 training.  It was my personal experiment on taking exam without any real life experience with vSphere 5. Surprisingly, it went quite well.

At the end of October I was finally notified on my last day of staying in Italy (which was 22nd of December) and only then I clearly realized that from my home country I would need to travel at least to Moscow just to take this exam. Assuming that I considered my chances to pass this exam lower than 50% I doubled my expected expenses for the exam. The final calculation showed that it might cost me about 2000 USD.  Therefore, I decided to take it in Europe, though I was very short on time.
At that moment  I had covered only 3 sections out of 9 from official blueprint. That's when I found another great VCAP-DCA guide from Edward Grigson. This guy has done stunning job. I think his guide is the fullest and the most comprehensive guide on the Internet right now. I reckon Edward should start selling it.

The bad thing about my decision to take the exam on 20th of December was overlapping schedule with my wife's ACCA exams. We have 2 children and that fact doesn't leave us, parents, much free time. Now we had to split that sparse free time between us. Moreover, we had to move to another country and I had to change a job which was making our situation even worse.We had to sell things, pack them, through them away, prepare documents for pets, find the proper school for my elder son, look for apartment, find new job, make a handover of all my working responsibilities to my colleagues. Guys, believe me that was a stressful mess.

The closest test center was in Zurich, which was about 3 hours of driving from Milan. The day before exam I checked weather forecast for Zurich, it was +9C which I considered safe to drive. I left home 6 hours before exam, early in the morning. In 1.5 hours, when I was somewhere in Alp mountains it started to snow. That was one of the hardest snowfall I ever saw in Italy. Well, weather forecast was totally wrong and I was on the fully snowed road in the car with summer tyres. I almost crashed couple of times and had to switch to first gear and drive at speed 20km/h for about 1,5 hour. Can't express how nervous I was considering that I had to sell the car the very next day and wasn't sure if could get to the test center on time.
Ok, I managed to get to the Zurich 1 hour before the exam. To make the things worse the fuse for cigarette lighter died so I knew my TomTom won't last more than 2 hours and getting home will be challenging. My head was full of bad thoughts about how the hell I am going to get back home.
Well, it is not the end of the story. I decided to take a quick lunch before exam. Ordered a salad with some poison-coloured sauce. When I was trying to get my wallet out I accidentally overturned the salad on my pants. At that moment I thought all these were just signs which were trying to say me I shouldn't have taken this exam.
When I showed up at the exam center I looked like some homeless guy - tired, nervous, with bright stains on my pants. Do you think it is the end of the story?  No way. When I sat at the computer and the lab started I discovered that keyboard had AZERTY layout. I thought I would be able to manage this problem, but first 15 minutes I was fighting my keyboard, particularly trying to find special characters which I needed to use in command line. I couldn't even type the password for vMA or Terminal session, instead I had to copy-paste it. Definitely, at this moment I was totally desperate and was already sure for 100% I would fail the exam. Luckily, the testing center administrator found another keyboard with QWERTY layout so we could swap keyboards and I could continue my lab in more familiar way. After 3,5 hours I still had 7 tasks uncompleted, but I was so exhausted and didn't have any will to fight. However, the rest of questions I fully answered and configured, though I wasn't sure I have done them correctly. So I had like 50/50 feeling about the results when I left.
I took exam on 20th of December, the results arrived on 13th of January. Not really 10 working days as VMware promises. My score was 350, but if it were even 301 I would be happy anyway. I didn't feel like I had strength to start it all over.

Practical notes:

1. VCAP-DCA guide from Edward Grigson contains almost everything you need to study about the lab. Just check what the difference is with the latest blueprint
2. I had to practice about 5-6 hours a day for about 7 weeks, non-stop.  
3. Very often preparing your lab to test some particular feature can take you 5 times longer than practicing that particular feature. In the beginning it was very annoying, but later on I recognized it helped me significantly on getting a lot of commands and actions on the tips of my fingers. 
4. If you have Linux command line experience you have serious advantage over the vSphere admins with Windows experience. I spent a lot of time studying basics of Linux.
5. I started preparations with virtual lab built on my powerful desktop at home, but then I got couple of spare HP blades at the office and also old EVA 3000. Honestly speaking, working with real hardware is a big advantage - it is faster and it is closer to real life (networking part for instance). At the end I used virtual lab very rarely. 
6. You have to practice every single feature mentioned in the blueprint. Although, in my lab I faced at least one thing that was not mentioned in the blueprint. Just read each section and question yourself if you can really configure it without looking in any VMware guide. Yes, you will be provided with vSphere documentation during the lab, but for the sake of saving time, imagine you won't have them at all. 
7. Leave the last week before exam to go briefly through all blueprint again.
8. Studying all blueprint doesn't mean you will be asked everything. You will be a bit disappointed that you are not asked to configure, for instance, vCenter Hearbeat even though you spent hours and hours studying the product you will probably never use. It is just matter of luck.
9. Memorize most popular Advanced Parameters of vCenter and virtual machines. I used some of them in real life, but never bothered to remember them. Most of the time I googled and then used copy-paste. When I had to find one during the lab I just wasted 5-7 mins without any luck.
10. The terminal access to the Lab is not fast. Switching between different windows takes time and sometimes is annoying. 
11. Some questions are not well worded. I had to re-read them several times before I could understand what they wanted me to do, but probably it is just me and my far from perfect English. 
12. Do easy questions first. If you see this question will take you some time to think about, make a note on your paper and go to the next one. 
13. The Biggest Surprise - some of the questions are insultingly easy. I mean, you spend days and nights to go deep into the vSphere technologies and then you are asked to configure something so simple that you think you misunderstood the question and there was some trick in it.
14. The Greatest Advice I read in one of the blog (sorry, can find the URL now) - write down questions' numbers in a row on the paper you will be given. This way you can track your progress by crossing out the questions you have completed and putting notes to questions you postponed. I am 100% sure if I hadn't read this question I wouldn't score the passing score.

It took me about 1 year and 5 month from the moment I first saw vSphere to the VCAP DCA certificate. If you don't have children yet, you are single, and full of passion for virtualization it shouldn't take you longer than a year :) 

Hope my story and notes can increase your chances or give you some guidance and probably motivation to pass this exam.

PS For those who are interested to know total number of people holding VCAP DCA certification - my ID is 553. At least I am in the first thousand. :)

If you find this post useful please share it with any of the buttons below.