Another budget vSphere home lab post - Part 1

A slightly more than a year ago I bought the top specs Macbook Pro (i7, 16GB, 512 GB) being totally sure it will be sufficient to run simple nested vSphere Lab. Honestly speaking I didn’t think through the requirements and what exactly I want to run in it, I just wanted a new toy. When vSphere 6 Beta was released it was a big surprise for me that 16GB of RAM is just enough to meet minimum requirements for 2 hosts and vCenter. I am not even talking about attempts to fit vSphere Horizon or vCloud Director setup in my nested lab. I used one quite powerful server at work for a while, but good time has passed very fast and it was definitely time to come up with a new permanent solution.

So here is the list of requirements I made at the very beginning of this journey

• Budget should be under 3000 AUD
• The lab has to be very quiet because having my own study is something to dream about so the hardware will be placed in a living room
• It has to look appropriate for a living room - I don’t like the idea of keeping rack mounted second hand DL360 under my desk. 
• The lab has to have descent performance so that I could run more than 10 VMs simultaneously.
• I know how fast vendors raise RAM requirements with every new generation of their products so I thought I would need a 2 hosts lab with at least 64GB of RAM in total.
• Power Consumption was not my highest priority, but I wanted my lab to work 24/7 so I would prefer to keep my electricity bills as low as possible.
• I didn’t want nested virtualisatoin, but to be honest I couldn’t find any particular reason why. Probably just had enough of it on my Mac and wanted to try something new.
• I wanted to have shared storage which I could also use as a home media library.

Frankly speaking, when I started looking through the myriad of options I thought it could take me at least a month to catch up on all new technologies - the choice of options is just huge. Provided that the last time I assembled a PC was about 15 years ago I felt very uncertain about success in my venture.

The first step was to understand what kind of VMs and how many I will run on the lab.

So I created very simple list of virtual servers, e.g. Domain Controller, SQL Server, vCenter, vCloud Director and put a rough estimate of resources needed for each VM. That helped me to define the minimum of RAM and Storage needed for the new lab.

Then I spent several days reading through tons of blogs and product reviews of everything - CPU, RAM, NAS, Case, etc and in the beginning the whole process was very chaotic, but by the end of the first week I started to understand what kind of lab I wanted and what my potential options were.

At first my attention was attracted to very neat and tiny computers like Mac Mini, Intel NUC and Gigabyte Brix. And as you can read in many blog posts a lot of people happily run vSphere home labs on them.

However, none of them can support more than 16GB of RAM and the models with Intel i5 and faster CPUs are not cheap. Also most of them come with 1 NIC only and no IPMI/ILO interfaces. Unfortunately, I had to say No to these very little and quiet devices.

Another option could be HP Microserver Gen8 or Gen9 as I have loyalty to HP server products, but these models are limited by 16GB of RAM too. That was a bummer as I really liked the mobile ILO application so that I could remotely check and restart my servers from my phone.

After getting rid of obvious mismatches with my goas I have come up with some possible choices.

So here were my 3 final candidates. 
*Please note that all prices are in AUD and are accurate as of July 2015:

Xeon build with Synology DS415+

This originally was my favourite. Just because word Xeon sounds to me like proper solution for virtualisation.

Pros:

• This particular model of CPU shows very descent performance.
• The motherboard has IPMI interface

Cons:

• The CPU TDP is 80W.
• The build was more expensive than others.
• 32GB of RAM is max amount supported by Xeon E3 CPUs.

Avoton Build with Synology DS415+

 

This is the setup I have ended up with. I guess the main reason to choose it was that the very idea of running virtualisation lab on Atom processors sounded so exciting to me. Unfortunately, not too many people used Avoton in their home lab so I couldn’t find any proofs with numbers that this Atom can handle the load of the lab.

Here are a couple of useful posts that gave me the idea of using Avoton for my lab - Rick Mur's blog and BeyondVM blog

Pros:

• TDP under 20W
• It is fanless  = quiet
• Very small form factor
• Supports up to 64GB of RAM
• Officially supported by VMware
• Has IPMI interface and 2 built-in gigabit NICs.

Cons:

• Benchmark results were not convincing. I wasn’t sure it will be fast enough.
• No VT-d support. That means you can’t pass through physical devices, e.g. like HBA or USB controllers directly to VMs. Howeer, I went through some lab setup I was going to deploy and none of them required VT-d.
• No EPT which means the ESXi host won’t be backing the Guest OS memory with large pages. On the other hand it means that TPS (RAM dedup) will be used by default.

Avoton build with whitebox NAS

Basically, I was trying to figure out if I could seriously cut my expenses by replacing Synology with some whitebox and XPenology/FreeNAS/Openfiler based on 4-core model of Avaton. 
The rest of the items on the shopping list are the same as in the build number 2.

Pros:

• 160 dollars cheaper than previous build
• Plenty of free solutions for NAS

Cons:

• No official support of Synology
• No hot plug hard drives
• No QuickConnect feature – allows you to provide very easy to use access to your Synology box from the internet without adjusting firewall rules or configuring VPN server

A couple of alternatives

One of the advice I received on Spicework forum was to get rid of Synology and use VSAN instead. I agree it sounds more interesting to build a lab using VSAN rather than being another guy with iSCSI storage. However, I also needed a solution for home media library so I couldn’t go this way. And also I can always play with VSAN later provided that I have space in my cases for 2 x 3.5 and 2 x 2.5 drives. Though I will need PCI card with extra NICs. So it is still an interesting option.

Another interesting option would be going with nested lab with SuperMicro SuperServer 5028D-TN4T. It is small, very powerful server that supports up to 128GB of RAM. The only problem was that I couldn’t buy it here in Australia and delivering it from abroad would cost me significantly more than I wanted to spend – the custom import fee, the exchange rate and delivery fee were the game changers.

A couple of words on the storage.

I knew that getting 6-8 bays NAS would be something over my budget (e.g. Synology DS 1815+) and that keeping a descent storage performance with 2 to 4 HDDs was something unreal so I thought I would use vFlash Read Cache to reduce the number of Read IOs hitting the iSCSI storage. 128GB SSDs are very cheap nowadays.

Also I went with 1TB HDD because I hope a year later I will replace them with SSD drives. So I didn’t want to pay extra for disk space. I will never use.

Synology was obvious choice as it has almost all features you may need for vSphere home lab – VAAI, Read and Write caching (provided that you have enough disk bays), LUN Thin Provisioning, LUN Snapshotting, compatibility with vSphere. And there are rumors Synology may bring deduplication with the realease of DSM 6.0 in a month. You can’t resist this list of features, can you?

I ended up with DS415+ which was most affordable model for my budget.

A hint on making a shopping list

There is a very useful web resource that allows to select all parts for your lab online and get them all checked for compatibility. Moreover, it even finds you the best price for each part http://pcpartpicker.com. There is an Australian version of it, but it wasn’t as useful as US web site as a lot of the parts are not sold in Australia.

When creating builds keep track of the prices and the websites where you found the items. I even went further and tracked the status of each item’s shipment and delivery.

In the next part of the post I will show you some photos of the build, we will talk about networking gear and the unusual feature of IPMI. 

And we will definitely look at performance of the lab on Atom CPUs. 

Feel free to ask me any question about this setup.