Table Of Contents
Introduction
Last year we published YCSB benchmarks that compared IBM SoftLayer with Amazon Web Services. This generated a lot of interest from lots of different folks in the cloud community. There was so much interest we decided to do it again with more platforms.
We reached out to an independent benchmarking enthusiast, Tim Callaghan from ACME Benchmarking, to run the tests for us. We compensated him for his time and work, but we took a hands-off approach to the actual benchmarking.
The nice thing about this kind of benchmarking is that we don’t care who wins. We’re not comparing Volt Active Data to a competitor, we’re using Volt Active Data to compare something completely orthogonal. We find this interesting; we hope you do too. We would love to hear your feedback, criticism and thoughts.
Recapping the Results
ACME Benchmarking ran YCSB workload B on Volt Active Data on six beefy servers, using six beefy clients running the Volt Active Data YCSB driver. This was done on four cloud platforms and five different kinds of instances.
You can find the complete write-up from ACME on their blog post. To summarize, the SoftLayer Bare Metal bested the full PaaS cloud offerings in performance (>1.5M ops/sec) and the price/performance (4.6B ops/$) for this specific benchmark. Google led amongst the full-cloud platforms in performance (>1M ops/sec) and price/performance (3.3B ops/$). Notably Google’s performance was twice as cost efficient as Amazon’s AWS.
Two Most Interesting Results
Bare Metal vs. Cloud Platforms
Most cloud platforms use virtualization technologies, such as KVM, to provide hosted Linux servers to customers. Virtualization technology has come a long way over the years, but it is still a layer between hardware and your operating system.
Comparing the SoftLayer Bare Metal performance and price/performance to the virtualized platforms is interesting. Perhaps it’s not surprising that bare metal does well, but the difference for this benchmark was significant: 50% better absolute performance and 40% more operations per dollar.
This result is also interesting because SoftLayer Bare Metal might be closer to what you can expect from similar servers your company may own and operate in its datacenter (or closet).
Competition Within the Cloud Platforms
The most interesting thing to us about the performance of the three leading cloud platforms is the performance of Google compared to the de facto market leader, AWS. Google was 1.8X faster in absolute performance and 2.1X faster in operations per dollar. It reminds me of the “AVIS: We Try Harder” slogan, where the second-place competitor has a lot more to prove. So here’s Google, the scrappy upstart?
Google also has a simpler pricing model than AWS, as described in the FAQ below. Overall, it appears Google’s cloud offering is worth evaluating, and the “Nobody got fired for choosing AWS” mentality might have legitimate cracks.
FAQ
Why YCSB?
We chose the YCSB benchmark because it is well known. It’s not a benchmark that requires much explaining on our part, and it allows comparison with other YCSB benchmarks out there.
The downside is YCSB is often very network intensive, so it’s going to stress some parts of a cloud platform more than others. This is pretty much unavoidable without comparing lots of varied workloads, which was outside the scope of this test. We hope to perform similar tests with other workloads in the future.
Why these platforms?
Most people agree that AWS is clearly the largest cloud platform out there, even though it’s not very public with specific usage numbers. Google Cloud and Azure are likely number two and three among platforms with virtualized Linux instances needed for software like Volt Active Data.
SoftLayer is the odd one here, but it’s really interesting because it offers bare metal hosting on an hourly basis. It’s clear from the results it has performance and price/performance advantages.
We’d love to look at more platforms, and may do so in the future, but focused on these four for this review.
Are these instances the same?
Ever go mattress shopping? You can’t find the same model at two different stores, even from the same manufacturer. This makes it hard to price compare. Ditto with cloud platforms.
ACME Benchmarking did its best to find instances that had 32 cores (however cores are defined), at least 60 GB of RAM, and 10 gigabit networking. This is a pretty good machine to run a super-high throughput Volt Active Data server, although some of our customers run slightly smaller instances. These instances ended up with different amounts of memory, disk and networking. Virtualized SoftLayer doesn’t even have a 32 core system available virtualized.
On the one hand, this makes the price/performance graph more interesting, as it normalizes the differences in instance types more than the raw performance measurement does. On the other hand, some of these instances provide a lot more memory and disk than others, and it’s sitting idle. The Azure instance had 448 GB of memory, which was mostly unused. ACME Benchmarking would have used an instance with less memory if one were available. Read the results with this in mind.
Note that the amount of memory or disk won’t affect Volt Active Data performance until the memory is actually exhausted. So this benchmark primarily compares CPU and networking performance.
How are these platforms priced?
Amazon charges by the hour, and offers discounts for one and three-year commitments, and for pre-payment. The difference in cost between hourly and pre-paying for three years is significant.
https://aws.amazon.com/ec2/pricing/
Google charges hourly, but discounts automatically as you use an instance longer. For example, once you’ve used an instance for more than 25% of a month, the hourly rate drops from that point on. It drops again at 50% and 75%. Thus the discounts are automatic.
https://cloud.google.com/compute/pricing
Azure offers fixed hourly pricing for most instances, but their larger instance prices fluctuate based on demand. Notably, Azure charges by the minute, rather than the hour.
https://azure.microsoft.com/en-us/pricing/details/virtual-machines/
SoftLayer charges less for monthly billing than for hourly billing. Monthly bare metal instances also offer many more hardware choices.
http://www.softlayer.com/info/pricing
Notes:
- Cloud platforms may charge additional fees for public bandwidth usage, and for any network-attached storage usage. Make sure your clustered servers are using internal networks to avoid billing surprises.
- In many cases with some or all of these providers, prices vary based on data center location.
- Several platforms also discount preemptable instances significantly, presumably to reduce hardware idling. However, preemptable instances are not very useful for Volt Active Data and other stateful system-of-record systems.
How can I use these results to inform my decisions?
It can’t be overstated that this is one benchmark, and your mileage may vary. As stated above, this YCSB workload stresses some parts of the cloud platforms more than others. This benchmark also doesn’t consider the ecosystems these platforms come with; it focuses only on the compute components within the platforms.
The most meaningful take-away is the understanding that both total performance and price/performance can vary dramatically between clouds, and also between virtualized platforms and bare metal platforms. If you’re trying to decide if it’s worth spending engineer time on considering a switch, these numbers might inform how much money could be saved.
Finally, the specific performance of the clouds, even though it’s one datapoint on one type of workload, is still pretty interesting.
Can I reproduce these results myself?
Absolutely. ACME Benchmarking used the community version of Volt Active Data, which can be built using public code on Github (https://github.com/Volt Active Data/voltdb). We expect the commercial edition of Volt Active Data that you can download from this website will perform identically if identically configured.
The YCSB benchmark driver is also open source and in the same repository (https://github.com/Volt Active Data/voltdb/tree/master/tests/test_apps/ycsb).
We used public clouds that can be used by anyone with a credit card. Note that a monthly commitment may be required to use SoftLayer Bare Metal.
If any of the configurations are unclear from ACME Benchmarking’s post, please let us know and we’ll add clarity as needed.
Join the conversation on Hacker News.
