Tuesday, May 13, 2008

Vista vs Vista SP1

We’ll start with our Futuremark benchmark applications, 3DMark 2006 and PCMark Vantage. 3DMark in spite of being a graphics benchmark is sensitive enough to pick up on any changes in CPU or GPU performance (in this case any optimizations that reduce overhead), while PCMark Vantage is a full suite of benchmarks to measure overall system performance. It’s also one of the few benchmarks that we’re using that has a 64-bit mode.

Starting with 3DMark, the change in performance is effective imperceptible at 0.2%, well within experimental variance. This just goes to show there haven’t been any changes in overhead. PCMark however is far more interesting; here we get a full 7% score increase, indicating that SP1’s effects are felt outside of our earlier benchmarks. Drilling down ino PCMark’s subscores, we found that the higher score is a result of improvements in data compression scores, data encryption scores, and searching in Windows Mail, all of which can be attributed to improvements in file I/O performance. All other subscores are virtually unchanged.

Moving on to our application specific benchmarks, we have our DivX encoding test, our iTunes/LAME MP3 encoding test, and the Retouch Artists speed test for Photoshop. DivX stresses I/O somewhat, while the rest of the tests are largely memory and CPU-bound.

Here we see no notable changes in performance moving to SP1. All of the application tests come back with virtually identical scores.

Finally we have our gaming tests. Games tend to be good a great way to stress all the components in a system, so this should give us a better idea of how far improvements in Vista’s file I/O system stretch. For our games we have the RTSes World in Conflict and Company of Heroes, and the FPSes Crysis and Unreal Tournament 3.

In spite of the more rounded nature of gaming tests compared to our application tests, the results are the same with no perceivable improvement in performance. At this point it’s clear that what performance improvements Vista does offer are limited to a handful of situations where we are specifically file or network I/O bound. What this means for any given application is that it is unlikely to see a performance improvement due to SP1.

We also ran some quick testing with startup and shutdown times to see if Vista improved performance there at all; there were a couple of hotfixes in SP1 that dealt with these matters.

Vista Startup/Shutdown Performance

Startup Time
33 Seconds
28 Seconds
Shtudown Time
32 Seconds
31 Seconds

While shutdown time doesn’t see any real performance improvements, we are surprised to see an improvement in startup time by several seconds. Vista is not notably slow to start up in the first place, so we weren’t expecting much improvement if there was to be any at all. Shaving off 5 seconds for a 14% improvement in startup time (getting startup below 30 seconds altogether) is a pleasant surprise.

Windows Vista SP1 review

Bug fixes are a big part of any Microsoft service pack, but not just for the bugs being fixed specifically by the service pack. Microsoft has released numerous hotfixes since Vista launched, correcting a number of issues declared significant enough that they need to be fixed before the next service pack, but minor enough that they’re not worth a full deployment and the kind of massive regression testing that entails. The result is that there are a number hotfixes already out that can potentially fix specific issues certain users are having, but because they aren’t well-tested they’re instead well-hidden with only a small number of users with extreme problems usually getting their hands on any given hotfix. Now that a service pack has arrived, Microsoft has rolled up all of these hotfixes into the service pack, in essence approving them for wide release and full support.

Among the 24 pages(!) of hotfixes that have been rolled into Vista SP1 are favorites such as the virtual address space fix and a fix for a conflict with NVIDIA’s USB controller and >2GB of RAM. Other additions include fixes for ejecting iPods, a fix for HybridSLI/HybridCrossfire (which is why the launch of these technologies is tied to SP1), and a fix for AMD Barcelona processors causing system reboots during Windows installations. While we could rattle off the entire 24 page list of hotfixes, the important thing to note here is that there are a number of small issues that have been “fixed” prior to SP1 but are only now being widely corrected. We’re going to spend most of our time going over the biggest and most noticeable fixes in SP1, but please keep in mind there are many more things addressed in this service pack than what we’re looking at today or are listed in Microsoft’s consumer-level product literature.

Among the most significant fixes to Vista in SP1 is Microsoft's work on further refining the User Account Control (UAC) prompts of Vista. Even after already being scaled down between the betas and Vista’s launch, these prompts are still rather prolific at times. An adjustment to the folder creation is the most prominently touted of these fixes, with the number of folder creation prompts (when creating a folder in a protected location) falling from four to one. Microsoft doesn’t list any further reductions in UAC dialogs, but as far as anecdotal evidence is concerned it certainly feels like they’ve done a bit more than that. This won’t change the public perception of UAC (or Apple jokes on the subject), but any reduction is welcome and perhaps will stem the tide of Vista users who are completely turning off this critical system feature.

Among all of the fixes in SP1, the biggest and most widely noticed will be the changes Microsoft has made to how Vista copies and moves files. There’s no two ways about it, Vista is slow at copying files - in fact frequently much slower than XP. Microsoft had a good reason for picking the methods they did for Vista but the payoff clearly hasn’t been worth it, so they’ve gone back to the drawing board and modified their file copy methods slightly to improve performance. Vista still won’t perform as well as XP (and we’ll get to why that is in a second), but with SP1 it’s definitely faster than it was under the original version of Vista.

The story of why Vista’s file copy speeds are slow so is a long and interesting one, with Microsoft’s tech blogging guru Mark Russinovich providing a particularly lengthy and descriptive explanation of the issue. If you’re curious as to what the specific details of the situation are, we encourage you to read the whole blog. For everyone else we’re going to take the liberty of paraphrasing and condensing it a bit, so that we can go over the changes to Vista without talking about things at such a low level.

According to Microsoft, there are three issues with the CopyFileEx method in XP that they wanted to correct with Vista.

  1. XP’s buffered file network operations resulted in the file being cached no fewer than 3 times: twice on a client requesting data and once more at the host sending data. This wasted a lot of memory, particularly on the client.
  2. Copying large files would eat up a great deal of memory when write operations couldn’t keep up with read operations (solid state drives in particular are prone to this) and the read data would be cached until it could finally be written
  3. File copying was a synchronous action that couldn’t be pipelined, resulting in poor performance over high-latency, high-bandwidth links; this is one of the areas Microsoft was working to optimize performance under Vista, which included optimizations at the networking level.