In this part of StarWind Virtual SAN review I will be testing and
comparing performance of different disk types using different combination of
features. But let me first start with a short description of my test bed.
- 2 x ESXi servers with Avoton CPUs - C2750DI4
- Each ESXI host has:
- 1 x 7.2 HDD
- 1 x 256GB SSD.
- 6 x 1GB NICs. All NICs are connected to a single TP-Link switch.
- First pair of NICs on each host is used for management and VMs traffic, another pair is dedicated for iSCSI traffic to Synology box and the last two NICs are reserved for StarWind servers.
- All VMs, including StarWind Virtual SAN Servers, run on a shared iSCSI storage (Synology).
- Each ESXi server hosts one StarWind Virtual SAN and has two local VMFS datastores - created on HDD and SSD disks
- The test disks, attached to Virtual SAN servers are placed on local HDD (data) and SSD (L2 cache). Thus, there is no other workload present on test disks.
- Each Virtual SAN server has two additional disks - one per each local datastore. That's where the test disks and corresponding Level 2 cache files are created.
There is
another important thing to mention before you get to the test results. I don't
have any intention to provide test results that could be valid for real-life
scenarios for the simple reason that my Lab setup doesn't meet the minimum hardware
requirements for StarWind, e.g. Intel Xeon E5620 (or higher) CPU.
My idea was to compare different combination of disks and
features and to see how each feature contributes to performance. For
instances, I was curious to check how many times Flat disk with L1 and L2
caches is faster than regular disk, or how HA Disks' performance depends on the
multipathing policy.
Here is all all
combinations I wanted to test summed up in one table
The plan was to get the baseline performance
running IOmeter natively in StarWind so that I could compare it with the
productivity of the Virtual SAN disks running on top of StarWind server and its
overhead.
Then I created 11 different StarWind devices as
per the table above and presented them to the ESXI hosts. I have also adjusted
the number of IOPs to 1 before ESXi hosts switch the path to Virtual SAN
servers as per VMware KB2069356. That means each host will be talking to both
Virtual SAN appliances (the local and the remote ones) when StarWind device is
mirrored to another appliance.
Then I have created a Test VM with 11 disks
placed on corresponding Virtual SAN datastores.
That's the final picture of my datastores
And
that's how the disks in Test VM look like
I hope
you get the idea of the test setup now.
IOmeter was chosen as a test tool, however, it
was taking IOmeter too long to create test file so I used RDFC (Random DataFile creator) utility to create test files.
Here is
the IOmeter test specification:
- 20GB Test File
- 4KB Block size
- 32
outstanding IOPs
- 100% Random, 67% Reads, 33% Writes,
- Duration 2 minutes.
The table below depicts the final test results using Round Robin multipathing
Now let’s talk about
performance results with regard to each disk type.
Flat
As you
can see Flat device result is very close to the baseline result. That means the overhead of
StarWind is minimal and can easily be neglected.
Flat-HA
Here you
can see benefit of Read Striping when ESXi hosts take advantage of all disk
spindles in both ESXi servers (387 vs 287). However, this positive effect
applies to Reads only. If you have Write intensive applications, it is better to
use Fixed multipathing policy.
Here is a nice article which also proves benefits of mutlipathing on StarWind
Flat-L1
L1 cache
proved to be pretty helpful as it lets Virtual SAN servers deal with IO bursts.
During the IOmeter test Virtual SAN cached all Writes in RAM while HDD served
Reads only. Once the test was stopped I could see Virtual SAN de-staging all
Writes onto HDD.
Here is how it looks in the performance graph
Flat-L1-HA
This
disk config reflects further performance improvement, again due to Read
Striping across both hosts.
While testing this particular disk type I found
out how important it is for StarWind product to have the correct IOPS limit for
Round Robin policy.
At first run I have default 1000 IOPS limit and
what I noticed that both hosts didn’t seem to cache any writes. As soon as
IOmeter started I could see that both Virtual SAN servers started to push
writes to HDD. It looked like L1 cache wasn't utilised at all.
This didn’t happen in previous scenario with standalone Flat-L1.
However, when I changed IOPS limit to 1 the caching started to work again.
Here is what, I believe, happens with IOPS
limit set to default value of 1000.
- ESXi host sends 1000 IOs over
the first path to the first Virtual SAN server.
- Writes are stored in L1
cache. Reads are serviced from HDD.
- ESXi host send another 1000
reads and writes over the second path to the second Virtual SAN server
- While second Virtual SAN
processes these IOs the first StarWind server thinks that workload is dropped
to zero and starts de-staging data stored in RAM.
- When ESXi host switches back
to the first iSCSI path the second StarWind server starts dumping cache data on
the HDD as well.
So you end up with the situation when both
Virtual SAN servers have plenty of cache to accommodate gigabytes of Write IOs,
but StarWind doesn’t take much advantage of it since it keeps data in the cache
for very short period of time and constantly flushes the cache data to HDD,
thus, impacting Read performance.
Therefore, with IOPS=1000 I was getting around
310 IOPS in IOmeter, but once IOPS limit was changed to 1 the performance
jumped up to 600.
Flat-L1-L2
Even
though the results are pretty good we can see that the presence of L2 cache
didn't improve results as much as I expected. I had a chat with StarWind
engineers and I was told L2 caching algorithm is going to be improved in the
next build. Also, it ought to be noted that my test runs were pretty short to
get enough data cached in L2 and to get significant advantage of it.
Flat-L1-L2-HA
Another example of benefits of Read Striping
with Round Robin multipathing. Also, if you compare this result with Flat-L1-HA
you can see that L2 cache hasn’t been useful at all. Again, let’s wait for a
newer build and hope SSD commodity disks contribute more into overall
performance.
LSFS
LSFS
shows serious increase in performance compared to Flat disk - 341 vs 287. This difference is valid only for
this particular test configuration. As it was mentioned before LSFS is pretty
good in dealing with Random Writes, but it isn't efficient for Sequential
Reads, for instance.
LSFS-HA
No
surprises here. The result is almost doubled compared to standalone LSFS (341 vs
654) due to double number of Virtual SAN servers and physical disks servicing
IOmeter.
LSFS-DD
Deduplication
increases performance of the LSFS even further (359 vs 341). This happens
because LSFS computes each data block's hash on the fly and will compare it
with hash table stored in RAM. If this data already exists on the LSFS disk
Virtual SAN will only have to update metadata table without writing actual data
which can save some Write IOPs.
Deduplication
applies to cache data too, which means more data can be stored in cache, thus,
increasing cache hits ratio.
LSFS-DD-HA
Again,
the performance is almost doubled by using Round Robin - 359 vs 690 IOPS.
LSFS-DD-L1-L2-HA
That's
the final test scenario where I wanted to see how Virtual SAN behaves with all
bells and whistles enabled. As you can see I could get amazing result of 740
IOPS on my very humble test hardware with data being mirrored between two 7.2K
HDDs only while keeping latency under 50ms.
I have also tested the HA enabled devices in Fixed mode just to see what kind of performance I get if I blindly follow best practices documentation.
I merged both - RR and Fixed - results into one table just to see what gives me the best results in my particular configuration
As you can see in all configurations the Round Robin multipathing was outperforming Fixed PSP. So, I need StarWind need to review and update their Best Practices documentation that says Fixed PSP is the only supported configuration on ESXi hosts.
I
know there are more device combinations and more combinations of IOmeter
specifications to try, but I have that much of time on these tests. If there is
interest in more test combinations and on different hardware I might write
another post on this.
Again, I would like to
stress out that these test results are given here for the sake of comparison of
features and disk types. The test system does not meet the minimal system
requirements and therefore test results are valid only for this particaluar
setup.
