Benchmarking Tiger vs. Panther

Benchmarking computer hardware has always been an arcane art, whether you're trying to measure modem throughput over real world telephone lines, printing speed of daisywheel printers using real text, or music ripping speed using real CDs.

Everyone wants to know how much faster the new stuff is than the old.

Some tasks are easier to measure than others. By working with the same set of music CDs, it's possible to compare the ripping speed of several different programs on the same computer - although most Mac users use iTunes by default and really don't seem to be on the lookout for something faster.

The biggest problem with benchmarks is that everything keeps changing. CPUs not only get faster in clock speed, but the underlying architecture changes. Different types of memory offer different levels of performance. SCSI, IDE, FireWire, USB 2.0, and Serial ATA each work a bit differently.

And software changes. With the release of Tiger (Mac OS X 10.4), Mac users also get new versions of Safari and Mail - and Apple has just released yet another iTunes update. "Real world" benchmarks that depend on iTunes, Safari, iMovie, Photoshop, and other applications can't compare directly with the same test using earlier versions of the program.

How can you compare last year's Power Mac G5, iMac G5, or eMac benchmarks under Panther (OS X 10.3.x) with this year's Power Mac, iMac, and eMac that ships with Tiger?

Macworld just answered that question with their Speedmark 4 benchmark suite. Their answer: You can't make a fair comparison; you need a new benchmark suite for the new operating system and applications. Comparisons with older Speedmark scores are meaningless.

To facilitate comparison between this year's hardware and last year's, they've installed Tiger on the earlier Macs and run Speedmark 4. With the 1.25 GHz Mac mini set as their baseline (arbitrarily given a Speedmark rating of 100), current Macs range from 90 (the 12" 1.2 GHz iBook G4) to 250 (the 2.7 GHz Power Mac G5).

The Mac mini Choice

Macworld used to pick one of the fastest Macs available as their baseline model, such as the Blue & White Power Mac G3. One has to wonder why they chose the second-slowest computer in Apple's line for the current revision.

My guess: Because it makes the top-end Macs looks so fast. After all, the 2.7 GHz dual CPU Power Mac G5 it two-and-a-half times as fast as the 1.25 GHz G4-based Mac mini.

Whatever their reasoning, the Speedmark 4 scores show an interesting picture. At the bottom of the Mac line are the G4 computers, with scores ranging from 90 to 135. Except for the eMac, these models also use 4200-5400 rpm 2.5" hard drives.

The next performance level is the single CPU G5 models, the iMac G5 and the 1.8 GHz Power Mac G5. Scores range from 141 to 160. These models also use 7200 rpm Serial ATA hard drives, which means they have both a CPU advantage and a hard drive advantage over the G4 Macs.

At the top of the ladder are the dual processor Power Mac G5 models, which have Speedmark 4 scores ranging from 196 to 250.

What We Can Learn

Faster CPUs lead to higher Speedmark scores. G5s score higher than G4s. Dual processor models score higher than single processor ones.

There are a few surprises. The 1.25 GHz eMac earns a slightly higher score than the 1.42 GHz Mac mini, although it only beats it in three performance tests. Why does it score higher? Probably because of the hard drive.

The 1.67 GHz PowerBook G4 outperforms the 1.6 GHz iMac G5 in a couple tests and scores nearly as well - a 135 score vs. 151 for the iMac. What that means is that the G5 processor itself is roughly comparable to the G4 at the same clock speed. For some tests it's better; for others, worse.

Two brains are better than one, but the overall Speedmark score only indicates about a 10% difference due to the second processor. (We'll know more when Macworld benchmarks the 2.0 GHz iMac G5 and can compare that directly with the 2.0 GHz dual CPU Power Mac G5. It's unfortunate that Macworld didn't benchmark the just-discontinued dual CPU Power Mac G5/1.8 GHz.)

Finally, some tasks benefit more than others from the G5 CPU and dual-processor design. Others are less influenced by these changes. For instance, iMovie rendering and the Photoshop CS2 test results have about a 2:1 range, while iTunes, MPEG2 compression, and Cinemark 4D have 3:1 to 5:1 ranges.

What We Can't Learn

My biggest disappointment is that Macworld didn't benchmark any dual processor Power Mac G4 model. It would be very interesting to see how the 1.42 GHz model compared with the 1.42 GHz Mac mini, which has a single CPU and a slower hard drive. It would also be nice to know how a dual processor G4 compares with the newer G5 models.

But most of all, we can't know how Mac OS X 10.4 compared to 10.3. For that we need to turn to a source that doesn't keep changing its benchmarks, such as Macs Only!

Macs Only! has been benchmarking Mac OS X on a Power Mac Cube since 10.1.5, and they've been doing a marvelous job of letting readers know how each system update performs in a wide range of tasks - and on a range of Macs.

Three of the graphics benchmarks have their best score ever under Tiger, duplicating a 700 MB folder is faster, but launching Classic is slower than it was under Panther.

Moving on to the Power Mac G5/2.5 GHz, the results are similar - but the Quartz Graphics benchmark was also slower than it was under 10.3.7-10.3.9. Then Macs Only! learned that Quartz 2D Extreme is disabled when Tiger is installed. When they enabled it, the score improved drastically.

Macs Only! tests 10.3.9 vs. 10.4 on three more platforms: a 1.25 GHz Mac mini, a PowerBook G4/1.67 GHz, and an iMac G5/1.8 GHz. With the exception of launching Classic, Tiger matches or outperforms Panther almost everywhere on the iMac, but results are mixed with the Mac mini and the PowerBook.

What We Learn

The big thing we learn from Macs Only! is that you can run benchmarks that compare different versions of the Mac OS. There are benchmarks that aren't dependent on a version of iTunes, iMovie, or Safari. These benchmarks may not be a "real world" as what Macworld does, but they let those of us with older hardware have some idea how Tiger will work for us.

The future belongs to the G5 processor and Core Graphics, a part of Tiger that needs at least 64 MB of video memory and a very recent graphics processor (Radeon 9600 or later). In the interim, the G4-based eMac was just upgraded and now has Core Graphics support, as do the current PowerBooks.

Missing out on Core Graphics support among current models - the iBook G4 (we expect an update soon) and the Mac mini. The single-processor Power Mac G5/1.8 GHz doesn't normally ship with Radeon 9600 graphics, but that can be included as a build-to-order option.

Although you'll get the best performance with modern hardware that supports Core Graphics, you'll also get good support for older hardware under Tiger.

As for those few places where Tiger is slower than Panther, I suspect Apple's software gurus are looking at them as they plan the OS X 10.4.1 update.

And you can count on sites such as Mac Only!, Bare Feats, and Accelerate Your Mac to tell you where 10.4.1 is faster and where it's slower than 10.4.0.