Foul! Unfair Benchmarks

4 September 1998. This article has generated an unprecedented amount of email. Most of it tells me that the iMac ships with graphics acceleration enabled; I had read otherwise. Another writer say the Power Mac G3 also ships with graphics acceleration enabled, so if NSTL tested it with graphics acceleration disabled, they had to do that intentionally. (Or they may have simply assumed it was disabled since they could find no way to enable it.) Several writers have also noted that the second page I quote from has been revised since I quoted from it, so some of the words I quote are no longer present.

On the one hand, I'm glad to learn that Apple ships computers with graphics acceleration on. On the other, I wonder why the Macs didn't benchmark better. And on the gripping hand, there's the issue of Microsoft Office using undocumented Windows code to optimize performance, something they can't do on the Macintosh.

And it looks like PC Magazine may be as reliable as ever.

I'm practically ancient for this industry. I remember lusting after the TRS-80 in Radio Shack flyers back in 1977. I think it was 1979 that I first put fingers to keyboard and used a personal computer.

In all the years since, computer magazines have been my faithful companions. In the early days, I read Byte, Creative Computing, and Compute! When I became a DOS geek, PC Magazine, Byte, and PC Tech Journal were my favorites. Today I lean toward the internet for the best and latest information, but still read Macworld, InfoWorld, PC Magazine, and others regularly. (And I am singularly upset with the folks who bought Byte magazine, put it on "hiatus,"and dropped almost every good writer they had. When it comes back - and it better, I'm a paid subscriber - it won't be the small systems cross-platform journal I've known and loved for almost 20 years.)

Computer magazines build their reputations on product test reports, which almost invariably include sophisticated benchmarks that look at the computer at both the component level and the application level.

MacBench, BYTEmark, and the ZD suite (CPUmark, Winstone, WinMark) are designed to fairly compare computers from different manufacturers, with different CPUs, with different subsystems. Best of all, BYTEmark is designed to do that across platforms, so it can evaluate a Pentium II v. an Alpha v. a PowerPC.

That may have been the reason for Byte's demise. BYTEmark showed the upstart PowerPC 750 (aka G3) outperformed the industry darling, the Pentium II. It also gave the Alpha a real run for the money. Best of all, the PowerPC did this with a lower clock speed, lower power consumption, and lower chip cost.

No wonder a Win-centric publisher bought out Byte.

But I am not here to condemn them, but to take TechWeb and PC Magazine to task for unfair comparisons between the Power Mac G3 and various Pentia.

John Gartner of TechWeb has been raked over the coals by the Macintosh community not only because he reported that testing by NSTL (National Software Testing Laboratory) showed PCs as faster than Macs (see his test report), but because a second page of the NSTL report clearly states, "The Mac tests were run in the default graphics mode. Enabling graphics acceleration mode would make full use of high-performance adapters such as the ATI Rage II+...." (emphasis added)

Of course, a lot of readers never get to the second page to discover that this was not a level playing field. NSTL was comparing graphics accelerated Windows machines with Power Macs that did not have acceleration enabled. That's why they carefully pointed out (near the bottom of the second page) that enabling graphics acceleration would have changed everything.

But to add injury to insult, Mr. Gartner followed up with Get Real, a column in which he said it is fair to compare computers as NSTL did.

Sorry, John, but in all my years of reading computer magazines, I've never heard of anyone knowingly testing a computer with an important feature like graphics acceleration disabled, especially if it shipped with the computer.

So what if it ships with acceleration disabled? Honest testers do their best to put every computer on a level play field, even ones from Apple Computer.

But that's not all

Now it seems that PC Magazine may have made the same mistake. In a series of tests released on August 28 (First looks: iMac, iMac v Pentium II: test results, iMac v Pentium II: what the numbers mean, iMac v Pentium II: how we tested), it looks like ZD Labs also neglected to enable graphics acceleration, instead running their tests in "default" mode.

This is the same magazine that used to test dozens of modems or video cards side by side, often tweaking settings to see how much performance they could wring out. I can't believe that today they'd rather just take equipment out of the box and test it without a single change. I thought we were dealing with geeks here.

It's a sorry day when testers at two leading PC labs can't be bothered to turn on the Mac's accelerated video and judge a Macintosh on it's true merits. Acceleration comes with the computer - all they have to do is turn it on.

Then again, that might make their favorite computers look bad.

Further reading

• Pentium v. PowerPC, MacKiDo
• Motorola and PowerPC (also covers 680x0 series), PC Magazine
• PowerPC v. Pentium II: escargot?, Mac Musings
• Great CPUs, past and present, John Bayko. See especially sections on 680x0, 80x86, PA-RISC, Sparc, Alpha, PowerPC, and Merced.
• RISC, CISC, and Your Mac, MacTips