Computer Benchmarks and Other Baloney: Don't Expect 2-4x Performance from Intel Macs

First published in Business in Vancouver/High Tech Office column

Benjamin Disraeli is famous for claiming there were three kinds of lies: lies, damn lies, and statistics. In the High Tech Office we may want to replace "statistics" with "benchmarks".

The idea behind computer benchmarks is noble - to have an objective test making it possible to compare different models of computer hardware independent from the software each is running. The problem is that no one runs a computer without software, so benchmark data offers at best a fun-house reflection of the real world.

On January 10th, Apple CEO Steve Jobs announced his company's first Macintosh models built using Core Duo processors from Intel. He boasted of benchmark tests showing the new models to be anywhere from twice to four times as fast as their PowerPC-powered predecessors.

While Jobs did the right thing of pointing out that real-world performance might not be quite so fast, most of the media ignored the fine print and highlighted the promise of super-speed.

The first Intel-powered Macs are now making it into consumers' hands, and we're getting to see the difference between benchmarks and reality.

I recently helped a colleague set up his new Intel iMac and was able to compare it to last year's (PowerPC-powered) model. A few things to note:

As the name "Core Duo" suggests, the new Intel CPU has two processor cores in a single unit. This promises twice the performance of a single CPU, which is reflected by the benchmark tests. But users will only see improvement on applications that have been written for multithreading, the ability to divide tasks between the two processors. Applications without multithreading won't see much, if any, improvement.

A computer processor's speed and power gets all the hype, but it's only one factor in the performance a user actually sees. Hard drive and memory access speed and video capabilities all affect real-world performance. The new iMacs have improvements in these areas, but nowhere near the 2x-4x speed improvements being touted.

For best performance, software code needs to be compiled into programs that are optimized for the hardware. Apple has reworked its operating system and many of its own programs to make good use of the new Intel processors, but most third-party software, including commonly used programs like Microsoft Office and Adobe Photoshop, have not yet been updated for the new hardware.

Intel Macs can run these older programs using built-in translation software called Rosetta, but the translation comes with a speed penalty. These programs feel sluggish, seeming to run at about half the speed they would on a recent-generation PowerPC iMac.

The result: The new hardware runs software optimized for it faster than the models they replace, but it runs other programs slower than last-year's models.

What's more, PowerPC Macs running the OS X operating can also run 1990s-era Mac software in so-called Classic mode. The new Intel Macs can't run Classic mode software at all.

Many Mac users, needing access to the odd Windows program, run a PC-emulator such as Microsoft Virtual PC; these emulators currently don't run on the Intel Macs. And while the Intel-powered Macs use the same sort of processors used in Windows PCs, it's proving to not be easy to get them to boot to PC operating systems like Windows or Linux.

In a few months, more software will support the new Intel-powered Macs; at that time, users should see the promise of blazing performance fulfilled. In the mean time, anyone buying a new Mac in order to see the super-speed promised by the benchmark tests is going to be sadly disappointed at the speed of their older software.