How Cloud A stacks up against the competition

We compared our performance using Redis-Benchmark next to some of the more traditional US based Cloud providers and this is what we found. Trying to keep the tests as “apples-to-apples” as possible, we chose the most similar instance sizes offered (1GB flavour) by each player and ran the tests using the same version of Ubuntu & Redis Server for each:chart_1-9

Cloud A vs. Digital Ocean vs. Rackspace vs. Linode
SET 315,556 234,907 74.44% 264,623 83.86% 64,259 20.36%
GET 312,793 256,344 81.95% 280,678 89.73% 68,939 22.04%
LPUSH 311,380 294,941 94.72% 350,185 112.46% 89,186 28.64%
LPOP 317,107 227,117 71.62% 316,509 99.81% 50,725 16.00%
Average: 80.68% Average: 96.47% Average: 21.76%

Cloud A Benchmarks:
Screen Shot 2014-02-03 at 10.17.32 AM

Rackspace posted their benchmarks here:
Screen Shot 2014-01-31 at 9.55.49 AM

Digital Ocean Benchmarks:
Screenshot 2014-02-03 10.13.16

Linode Benchmarks:
Screenshot 2014-02-03 09.42.17

For Digital Ocean & Linode we created instances of the same size and the ones we tested for Cloud A. For the tests we ran, we took 5 samples for each test, removed the highest, lowest, and then took an average of the 3 remaining results.

What does a high benchmark score mean? The Redis team talks about the benchmarking tool here, and the factors that impact the requests per second the most are: virtualization overhead, per-core cpu performance, and RAM speed to name a few.

We are always measuring our own performance and taking queues from competitors on areas to optimize. We also realize that single benchmarks are to be taken with a grain of salt and we do not claim to be redis performance experts.

The best metric to consider is end user experience, in a recent video demo of how to create an instance it only took 24 seconds for Cloud A to instantiate a standard Windows VM (you can see it here) whereas Digital Ocean proudly advertises it takes 55 seconds to do the same think on their platform.

There will be more benchmarks posted soon around I/O performance, one of the biggest challenges in the public cloud space.