You'll get your Mac news here from now on...

Help TMO Grow

Subscriber Login

Advertising Info


The Mac Observer Express Daily Newsletter


More Info

Site Navigation

Home
News
Tips
Columns & Editorials
Reviews
Reports
Archives
Search
Forums
Links
Mac Links
Software
Reports
Contact

Visit Charle Gaba's new System Shootout site for the latest shootouts!


Desktop Shootouts
New!
(U.S. $):
Apple Desktop Lineup
Laptop Shootouts
(coming soon!)
$800
Laptops
$1,100
Laptops
$1,600
"Mini-Me's"
$2,000
Laptops
$2,600
Laptops
$3,000
"LapZillas"
Apple Laptop Lineup

Issues with Benchmarking
by Eric H. Yang

A widespread problem in the computer world right now is the tendency to run benchmarks to determine the superior CPU/VideoCard/etc. The whole notion of what benchmarking is has been lost on the online community and many flame wars have started based on incorrect perception of what benchmarking really is.

Benchmarks cannot be used as an accurate basis of comparison between Macs and PCs. Benchmarks test only how fast a program runs on a specific machine and nothing more. Many people think that because Platform A runs Benchmark A faster than Platform B, then Platform A is in fact faster. The only answer that a benchmark gives you, is that Platform A runs faster than Platform B on this current program. A benchmark can easily be changed to show one platform in a better light than another platform. An example of this is Intel's investment in the benchmarking firm BAPco. In their investment, they hope to get benchmarks that show that their CPUs are the fastest CPUs on the market today. However, just because Intel's processors score higher than other processors in BAPco does not mean that Intel processors are faster, it just means that Intel's processors are faster at running BAPco benchmarks.

This leads many people to judge various benchmarks as unfair because they feel that it gives an advantage to one processor over another. However, optimizations are an important part of the development process. If the software developer knows that the majority of his clients are using a specific hardware setup, then it would be logical for him to optimize it for the hardware setup that the majority of his users would be using. Therefore a benchmark optimized for whatever platform he was developing would give a better picture as to how fast his programs would run comparatively on the different platforms.

In an attempt to alleviate this problem, companies have come up with real world benchmarks as opposed to synthetic benchmarks. These benchmarks run tests based on real world code, code that has been optimized to run as fast as possible on a wide array of different machines. However, the tendency is to run one benchmark and come up with a statement such as, "Since this program is floating point intensive and runs faster on Platform A, Platform A has a better floating point unit." However, due to the scenario presented before, optimizations may have favored one platform over another and perhaps another program may run floating point calculations faster on the other platform.

When Steve Jobs presents benchmarking information that shows the Mac is faster than PC, or when PC enthusiasts present benchmarking data that shows otherwise, the reader should be wary as to what they are testing. In the majority of the benchmarks, the PC users are presenting data that is irrelevant to the Mac user, and the Jobs is presenting information that is irrelevant to the PC world. For example, a graphical artist should not be concerned as to how fast a PC can run Quake 3 vs. a Mac but rather how fast the programs he uses such as Photoshop or Illustrator perform specific tasks run on each platform. Likewise a gamer should not care how fast Photoshop runs on his machine compared to another platform, but rather how fast it runs his favorite games. If the point of showing benchmarking data is to convert users from one platform to another then, both parties are failing miserably because of their failure to present relevant benchmarks.

The obsession with speed has clouded over the primary reason people get one type of computer over another. Since Mac apps can't easily be recompiled to run on a PC, and vice versa. People want to be able to communicate easily with their peers and usually the easiest way to do that is to run compatible platforms. Comparing speed between the two platforms is not a very useful thing when it comes to picking a platform. Until a Unix type arrangement has been made where both platforms have identical file systems and applications can be ported over easily by the end user, the comparison in speed is useless.

The information above is as accurate as I could make it based on a reasonable time studying the respective companies' Web sites. There is no guarantee that these configurations, prices, and features are the best that one could come up with; assorted bundles, free internet access, free extra memory, and so on can most likely be found for any of the above systems. The point of this chart is to demonstrate that Apple's lineup holds its own against any brand-name computer, even in areas in which it has traditionally had resistance. In short, do not base your purchasing decisions solely on this chart or information. Neither I, nor The Mac Observer will be held responsible or liable for any purchase decisions you make based on this presentation.

Charles Gaba started the AAPLTalk System Shootouts at AAPLTalk.com, and has brought them to TMO. Charles is the creative talent behind Brainwrap Web Design.

AAPLTalk System Shootouts Archives

Back to The Mac Observer For More Mac News!



Today's Mac Headlines

[Podcast]Podcast - Apple Weekly Report #135: Apple Lawsuits, Banned iPhone Ad, Green MacBook Ad

We also offer Today's News On One Page!

Yesterday's News

 

[Podcast]Podcast - Mac Geek Gab #178: Batch Permission Changes, Encrypting Follow-up, Re-Enabling AirPort, and GigE speeds

We also offer Yesterday's News On One Page!

Mac Products Guide
New Arrivals
New and updated products added to the Guide.

Hot Deals
Great prices on hot selling Mac products from your favorite Macintosh resellers.

Special Offers
Promotions and offers direct from Macintosh developers and magazines.

Software
Browse the software section for over 17,000 Macintosh applications and software titles.

Hardware
Over 4,000 peripherals and accessories such as cameras, printers, scanners, keyboards, mice and more.

© All information presented on this site is copyrighted by The Mac Observer except where otherwise noted. No portion of this site may be copied without express written consent. Other sites are invited to link to any aspect of this site provided that all content is presented in its original form and is not placed within another .