Apple Exec Responds To Benchmarking Controversy

by , 9:00 AM EDT, June 25th, 2003

Controversy has reared its boisterous head, especially in the online geek communities, over Apple's claim that the Power Mac G5 is the "fastest personal computer" on the planet. The controversy comes from Apple's testing methods to get benchmarks that support that claim, which some have said were grossly unfair to the Intel world. The company published the testing methods, and Mac and Wintel advocates alike have weighed in on the issue. For more information on this situation, see TMO's full commentary on the subject.

In a somewhat rare move, an Apple executive has commented directly on the subject. In a recent phone interview posted to Slashdot, Apple's Vice President of Hardware Marketing Greg Joswiak claimed that Apple's G5 and P4/Xeon comparisons were designed to be as fair and balanced as possible, disclosing all details in an effort to prove this fact. Joswiak also said that if Apple were trying to be deceptive, releasing all of the test data would be "a silly way to do things."

He goes on to explain Apple's choice of the GCC compiler on both platforms, instead of using more platform-optimized compilers, such as Intel's ICC. According to Joswiak, the Intel-based machines would fare better with the Intel compiler, but the G5 would score higher with another compiler also. Also, according to Mr. Joswiak, Apple chose the options that made the Intel chips look faster, such as running the tests under Linux instead of Windows, and turning hyperthreading off. From the Slashdot article:

He said Veritest used gcc for both platforms, instead of Intel's compiler, simply because the benchmarks measure two things at the same time: compiler, and hardware. To test the hardware alone, you must normalize the compiler out of the equation -- using the same version and similar settings -- and, if anything, Joswiak said, gcc has been available on the Intel platform for a lot longer and is more optimized for Intel than for PowerPC.

He conceded readily that the Dell numbers would be higher with the Intel compiler, but that the Apple numbers could be higher with a different compiler too.

Joswiak added that in the Intel modifications for the tests, they chose the option that provided higher scores for the Intel machine, not lower. The scores were higher under Linux than under Windows, and in the rate test, the scores were higher with hyperthreading disabled than enabled. He also said they would be happy to do the tests on Windows and with hyperthreading enabled, if people wanted it, as it would only make the G5 look better.

More interesting technical information can be found in the article on Slashdot. Greg Joswiak will be presenting the Macworld New York keynote address in July.

The Mac Observer Spin:

It is refreshing, exciting even, to see Apple actually respond to criticism in a direct and open manner. There is little doubt that this discussion will continue, and Mac bashers are not going to be swayed by these comments from Mr. Joswiak any more than Mac zealots are going to look objectively at the situation.

Be that as it may, the fact is that Apple is being very open about this. From publishing the testing methods, a rarity in the world of benchmarks, to explaining the company's reasoning, as Greg Joswiak did, to supposedly being willing to have additional tests run that meets the desires of Apple's critics, the company is facing its claims and its critics head on, in an open fashion. That's a Very Good Thing™.

Here is how this is going to play out: The lemmings are going to remain convinced that Apple has cheated, no matter what is said, done, or shown. That just isn't going to change. Apple's claim represents a direct challenge to the belief system of some of these people, as strange as that may be, and that belief system is that Macs suck because they are slower. This is a subject far too complex for a TMO Spin, but the long and the short of it is that those who prefer to be on the inside, looking out, want to squash all dissent because they are threatened by that dissent. It is those people who will cling to the notion that Apple cheated no matter what.

On the other side of the fence are Mac zealots who will brook no criticism of Apple under any circumstances. They would cling to the idea that G5s were faster than Deep Blue if Apple had said so.

In between are those actually interested in the reality of this situation. Is a Power Mac G5 the fastest PC on the planet? Is it not? Is it at least just as fast a Intel's offerings? Has the speed title been turned into a race again? Those people are not going to really get their answers until Apple ships these machines, and lots of real users get to use them. It doesn't much matter what kind of benchmarks Apple, Dell, Intel, or IBM release, because it is real-world uses that matter.

Apple did include some real-world tests during Monday's WWDC keynote, and those real-world tests showed the G5 kicking some proverbial butt. Our advice to those following this is to wait until people outside of Apple (or any corporation involved in making computers and software) can take hardware and software available on the street and run their own tests. That's when we'll truly know.

In the meanwhile, the Mac bashers and Mac zealots can have their screaming match. For our money, we wish a pox on both their houses.

Slashdot