Benchmarking Parallels, Fusion, and VirtualBox Against Boot Camp

| Analysis

VM Benchmarks Parallels vs Fusion vs VirtualBox vs Boot Camp

A few weeks ago, we provided a look at the major options for Windows virtualization on Mac OS X: Parallels Desktop 8 and VMware Fusion 5. A head-to-head comparison showed that Parallels 8 beat Fusion 5 overall, although performance was very close in most benchmarks.

Many readers asked us to perform additional testing to determine the improvements between Parallels 7 and 8, and how the free VirtualBox software compared to the paid options. Readers were also curious about native Windows performance in Boot Camp.

We’ve spent a few days testing and we now have our results. Read on for the ultimate performance comparison between Parallels 7, Parallels 8, Fusion 4, Fusion 5, Virtual Box, and Boot Camp.

Hardware & Software

Our tests were performed on a 2011 27-inch iMac at 3.4 GHz with 16 GB of RAM, running OS X 10.8.1. We used Fusion 4.1.3, Fusion 5.0.1, Parallels 7.0.15107, Parallels 8.0.18108.797180 (we have to talk to Parallels about a simpler versioning scheme), and VirtualBox 4.2 to virtualize Windows 7 Professional 64-bit.

All virtual machines were stored on and accessed from an external Pegasus R4 Thunderbolt RAID array with four 3 TB 7200 rpm hard drives in a RAID 5 configuration. We could not install Windows via Boot Camp on this drive, so native Windows was installed on the iMac’s internal 240 GB OWC SATA III SSD.

All virtual machines were configured to use four of the iMac’s eight logical processors and 4 GB of RAM. The configuration options don’t completely translate between applications, but each was set up to maximize the performance of the virtual machine over that of the host OS.

For the Boot Camp tests, all benchmarks when possible were set to use only four cores to provide a more accurate comparison.

Testing Methodolgy

Except where otherwise noted, each test was run three times in each configuration and the results, as long as they were within five percent of each other, were averaged. If any result was abnormal, we retested until we could determine the source of the abnormality and then discarded that result. Not all tests were possible in all configurations. These exceptions are noted for each test.

PCMark 7

Futuremark’s PCMark benchmark attempts to evaluate overall system performance, and includes tests for computational tasks, image and video playback and manipulation, web browsing, gaming, and storage speed.

VM Benchmarks PCMark 7

It should first be noted that some virtual benchmarks, such as the Lightweight and Productivity tests, report artificially high scores. This is due to the inability to have an "apples to apples" CPU comparison between virtual and native hardware, and to certain virtualization optimizations that can inflate synthetic benchmark scores.

Also of note is the Boot Camp result for the Computation test. PCMark 7 does not allow a user to restrict the number of CPU cores, so the Boot Camp tests were able to use 8 logical cores while the virtualized tests were restricted to 4. Normalized Boot Camp performance would still be better than virtualized performance, but not by as large of a margin.

Other than those tests, Boot Camp is the obvious and expected winner, although virtualized performance is not far behind in some categories. As we demonstrated in our previous testing, Parallels 8 offers the best virtualization performance for this test, with Parallels 7 in second place and Fusion 4 and 5 nearly tied for third. The free VirtualBox has a long way to go to match the performance of the commercial software applications and comes in a distant fifth place among virtualized options.

VM Benchmarks PCMark 7 Overall

If we focus just on the current versions of the virtualization software compared to Boot Camp, we can see that Parallels 8 holds a roughly 10 percent advantage over Fusion 5, while native Boot Camp tacks on about 21 percent higher performance over Parallels 8.

3DMark06

Futuremark’s 3DMark06 is a DirectX 9 gaming benchmark that attempts to stress a system’s GPU and CPU the same way that a highly detailed game would. It’s a bit old at this point but it provides one of the best ways to test DirectX 9 performance, for which both Parallels and Fusion offer full support. VirtualBox has basic 3D support but would not run the 3DMark06 benchmark.

VM Benchmarks 3DMark06

The results show the same trend found in the PCMark 7 tests: Boot Camp takes first, Parallels 7 and 8 lead the virtualization results, and Fusion 4 and 5 finish last.

VM Benchmarks 3DMark06 Overall

Again focusing just on the current versions of virtualization applications, we see that Parallels 8 scores about 8.5 percent higher than Fusion 5, and about 11.5 percent lower than Boot Camp.

3DMark Vantage

Futuremark’s 3DMark Vantage is a DirectX 10 benchmark. While the iMac’s GPU supports DirectX 10 natively in Boot Camp, only Parallels 8 supports the multimedia API while virtualized. Parallels classifies its DirectX 10 support as “experimental,” so improvements in performance should be expected over time.

3DMark Vantage has various presets for its CPU and GPU tests. We used the default settings for the Performance (P) and Entry (E) tests.

3DMark Vantage VM Benchmarks

The results show that DirectX 10 support has a long way to go. While it is functional, gamers should not expect to be virtualizing their favorite DX10 games any time soon.

Geekbench 2.3.4

Geekbench is a multi-platform tool for measuring a system’s computational and memory performance. It does not test graphical or storage capabilities, but is useful in that it scales from systems as small as an iPhone to those as large as dozen-processor workstations.

Geekbench can be run in either 32- or 64-bit mode. As we were using a 64-bit version of Windows with 4 GB of RAM, we ran the 64-bit mode.

Geekbench VM Benchmarks

Again, Parallels has the advantage over Fusion in most tests, although both applications come close to native performance in the memory and stream tests. The free VirtualBox holds its own in this test as well. It doesn’t win any category, but it does keep within ten percent of Parallels on most tests. Boot Camp holds substantial leads in Integer and Floating Point calculations.

VM Benchmarks Geekbench Overall

An examination of the overall Geekbench score for the current versions of virtualization software tells the same story as previous tests and ranks performance in the following order: Boot Camp > Parallels 8 > Fusion 5 > VirtualBox 4.2.

Cinebench 11.5

Cinebench is a multi-platform benchmarking utility that is based on Maxon’s Cinema 4D rendering software. It tests OpenGL graphics performance and multi- and single-CPU rendering capabilities.

Like the GPU tests above, VirtualBox could not run the OpenGL portion of the Cinebench test. It did run the rendering tests, and so it is included in the chart with a zero for its OpenGL results.

VM Benchmarks Cinebench

Parallels 8 scored the highest of the virtualization applications on the OpenGl test with 41.4 frames per second, but all virtualization software fell well behind native performance at 74.65 frames per second.

It was a much tighter race in terms of rendering performance. We limited Boot Camp tests to four processors but performance was still slightly better than virtualized options. Parallels 7 held a surprising, albeit slight, lead among virtualization software and VirtualBox scored very well compared to its commercial peers, taking second place in the multi-core test.

Just Cause 2

Just Cause 2 is an open-world action game released by Eidos Interactive in 2010. It supports DirectX 10 and so, like our 3DMark Vantage test above, was only tested on Boot Camp and Parallels 8 via its experimental DirectX 10 mode. We tested four resolutions on the “Desert Sunrise” built-in benchmark.

VM Benchmarks Just Cause 2

Parallels 8 offers playable frame rates, but falls far short of native performance. There also appears to be some issues with Parallels’ DirectX 10 driver, as resolutions between 1440x900 and 800x600 scored roughly the same. As we advised above, gamers should not yet plan to enjoy DirectX 10 gaming from this generation of virtualization software.

Crysis

Crysis, released by Crytek in 2007, was once the ultimate measure of a gaming PC's performance, bringing even the most expensive and capable machines to their knees. In a sign of how far we have progressed in the past five years, Crysis is playable in a virtualized environment on a Mac. Competitive gamers who demand at least 60 frames per second will be disappointed, but more casual gamers who want to see if their Mac “can play Crysis,” will be able to enjoy the game.

Using the Crysis Benchmark Tool, we ran the medium quality test in DirectX 9 mode at three resolutions.

VM Benchmarks Crysis

Boot Camp has a huge performance lead but a surprising second place result is Fusion 4. As noted below the chart, however, there were noticeable graphical glitches during the test. It was still playable, but considering the glitches and the performance of the other applications, Fusion 4’s results do not carry much weight.

Between Parallels 8 and Fusion 5, results were nearly identical with the exception of the 1280x720 resolution, where we think Fusion hit a virtualized processing limit, as its score is the same as 1680x1050.

Mafia II

Mafia II, released in 2010 by 2K Games, is a third-person open-world adventure game. It runs in DirectX9 but would not run properly on Fusion 4 or 5, either crashing before completing the benchmark or freezing and stuttering throughout the run. We have therefore omitted those applications from the chart.

VM Benchmarks Mafia II

Again, Boot Camp provides substantially better performance, but the game does run at playable frame rates in Parallels. Parallels 8 increased frame rates by about 12 to 15 percent over Parallels 7, although we think there is a bottleneck in the way Parallels virtualizes the GPU, as frame rates were roughly the same across the bottom three resolutions.

Boot Times

Using a clean installation of Windows 7, we timed five cold boots for each application with a stop watch and then averaged the results, rounding to the nearest second.

The timer started when we clicked the power or start buttons to launch the virtual machine from within each application, and it ended when all items in the Windows system tray were loaded.

VM Benchmarks Cold Boot Time

Parallels 8 scored the fastest boot time at 19 seconds, followed by Fusion 5 at 22 seconds. Parallels 8’s time represents a 42 percent increase over Parallels 7.

VirtualBox took the longest to boot, at 38 seconds, nearly twice the time as Parallels 8 and Fusion 5. As long as users plan to keep a single virtual machine open for a while, however, 20 additional seconds of boot time should not severely impact a user’s experience.

Conclusions

In light of our recent tests, there should be no surprises in today’s results. Users who need the absolute best performance should natively boot Windows using Boot Camp. Those who want a compromise between performance and convenience should consider Parallels 8 or Fusion 5, although Parallels 8 holds a performance advantage in most tests.

Users looking for a free solution can try the free VirtualBox, although its significantly lower performance underscores the adage that “you get what you pay for.” Depending on the desired task, however, VirtualBox may be all some users need.

It should be noted that these tests look only at raw performance. Both Parallels and Fusion have a bevy of additional features, many of which are exclusive to one application or another. Raw performance is not the only factor that should be considered when choosing a virtualization application, and users should check out each application’s feature list to see the absence of certain features are deal-breakers for their usage needs.

In the end, Parallels 8 and Fusion 5 improved on the performance of their direct predecessors, and Parallels 8 wins in overall performance.

Users interested in either application can download free trials from the Parallels (14 days) and Fusion (30 days) websites. VirtualBox is available for free from Oracle. Boot Camp is included in OS X. All applications can be used to virtualize many different operating systems including free distributions of Linux, although those looking to virtualize Windows will need to obtain a physical installation disk or disk image of the OS.

Comments

geoduck

Thank you. VirtualBox is generally ignored and so there hasn’t been much information about how it compares to the other solutions.

Yes, it’s not quick. Primarily I use VB with Win7 to access Windows Based remote desktops and servers with the heavy computing done on the remote system. As a result VB is fast enough for my needs. I have noticed the slow boot times though. Being that I run ‘real’ Win7 systems at work I know how fast they can get going, and no question about it, VB is not quick. I wouldn’t use VBfor audio or video work or cutting edge games, but it fits my needs.

KitsuneStudios

Thanks! This is incredibly useful. It’s amazing to see just how well Parallels and Fusion have become in catching up; for my money, those numbers seem well worth the convenience.

Unfortunately, I use Boot Camp almost exclusively for gaming, so these aren’t worthwhile purchases, but that’s great to know too. wink

B

It’s somewhat unsurprising native via Boot Camp performed so much better against virtualization and I question the gap in these numbers, given the BC tests were carried out on an SSD with a different performance spectrum from the RAID5 array.

Virtualbox is also the only free option here.

Also, boot time is moot when you can restore a snapshot state in <5sec. rather than booting the OS from scratch.

archimedes

Sure it’s slower… but I think it’s pretty amazing that you can now run Crysis at 1080p at nearly 30 FPS in a virtual machine on an iMac!

Jamie

I use Parallels for video game console emulation and it works great - it’s support for devices like game pads etc. is fantastic, and In it’s cohesion mode it is almost invisible - I can’t imagine anyone with more modest needs such as web browsing / testing or Office capabilities wouldn’t be perfectly happy with it. I’ve used Virtual Box for lighter tasks as well and have found it more than adequate if graphics or video performance isn’t a necessary part of the equation. For natively Windows-centric games, I’ve actually had tremendous success with WINE (though you’ll need a very fast Mac and one should definitely know their way around the command line for it - but nothing beats not having to install Windows in the first place!).

I think it’s amazing too! Looking back to the Virtual PC or Orange PC days, it’s like cavemen discovering fire. wink

Marco

Both Parallels and Fusion can be attached to a bootcamp partition.
Are there differences when using Parallels/fusion on a bootcamp partition against using it with a own virtual disk?

Jan-Willem Arnold

How much integration do you really want? This is where Parallels really is missing the point. When I ctrl-click on a movie I now get the option to play it in Quicktime (OSX) and Quicktime(in a Windows, Parallels virtual machine). I use OSX as my primary system. I really dislike the way Parallels gets into my system without even asking, and even without an option to disable it. An ‘open in IE’ button in Safari? Nonsense. Parallels looks like you bought the wrong computer in the first place. When you uninstall Parallels those bits remain in place. You have to search the Parallels forum and type in terminal commands to remove them. And even then, bits and pieces remain left. Some piece of advice: think before you decide to evaluate Parallels.

Running OpenSuse was quite a challenge in Parallels. Video support was really bad. Again you had to go to the forum and do a lot of manual steps on the command line to fix this.

In fact this is some deja vu from that other piece of software of Parallels known as Plesk. That also disobeys general architecture and design principles and intrudes into your system to a level far beyond what you should ever want. Because of that it is known to break in system updates and I fear that for Parallels as well.

I have been using VirtualBox over a long time. One area where VB shines is disconnecting external drives where the guest OS resides. Even without stopping the virtual machine first (oooppps) it often just gets paused by VB and you can restart it after reconnecting the drive (when you are back at your desk again). Of course Germans would never do such a thing, and indeed some tests in Parallels just let to crashes of the guest OS.

For me a guest OS should remain what it is: a guest. A place where I can test things, do development and what will remain isolated to my primary system. It is of course nice to have a coherence mode and to be able to copy paste. But I want to avoid any risk of mistakenly do things on the wrong system and I want to be able to dump a guest OS at anytime without consequences to the main OS. And I will never, never, never want to see an ‘open in IE’ button again. Parallels is not the way to go.

It's Magical

Your conclusion should have read:

Users who need the absolute best performance should natively boot Windows on a PC.

You also should have run the VMs on local storage since not many would carry a DAS device with them.  You could also have configured a PC with the same specifications to show the performance difference.

Log-in to comment