Wednesday, August 20, 2003
Is Windows Mobile 2003 Really Faster?
Posted by Jason Dunn in "THOUGHT" @ 09:05 AM
I had the rare opportunity to run Spb Benchmark on a single iPAQ 3650 running three generations of Microsoft's Mobile operating systems (the Pocket PC 2003 image for the iPAQ 3650 came from Microsoft - it a special testing-only build that won't be released to the public). This gave me the metrics I needed to make this definitely statement: Windows Mobile 2003, the first Pocket PC operating system built atop Windows CE .NET 4.2, is the fastest all-around operating system we've seen so far on these little devices we enjoy using so much. I present to you a series of benchmarks that show you exactly how much faster it really is! The Pocket PC 2002 and 2003 were performed on the same device, while the Pocket PC 2000 tests were performed on a different device, but of the same model.
First, let's look at how the Windows Mobile 2003 iPAQ compares with the baseline index, which is based on a Pocket PC 2000 iPAQ 3650. Take a look:
• Overall, Windows Mobile 2003 is 15% faster
• The CPU performance of the StrongARM 206 Mhz is 25% faster under Windows Mobile 2003 than under Pocket PC 2000
• The file system is slower (0.3%), but only slightly so in the overall score (the new OS is much slower in some key tests you'll see later on in this article
• The graphics subsystem of Windows Mobile 2003 is vastly improved over previous versions - a whopping 40% faster.
• ActiveSync has the most dramatic improvements - an incredible 103% boost in speed over Pocket PC 2000.
Read on for the comparisons between Pocket PC 2000, 2002 and 2003 - I think you'll be surprised at how much of a speed drop we endured with Pocket PC 2002 in some tests, and what a dramatic difference there is in moving to the 2003 OS. Clearly, the resources required to move to the new Windows CE .Net 4.2 core were worth it, even if it meant that we didn't get that many new application-specific features in Windows Mobile 2003.
The screen shots below do not represent every possible test that Spb Benchmark is capable of performing, only the ones I felt were more relevant and showed the most dramatic performance differences. The indexes and averages are based on the entire suit of Spb Benchmark tests.
Figure 2: We lost speed with Pocket PC 2002, but gained it back (and then some) with Pocket PC 2003
Figure 3: Windows Mobile 2003 offers significant CPU enhancements for any ARM processor, without any special XScale tweaks. This is further proof to me that Microsoft made the right decision in not making special optimizations for the XScale processor, and instead focusing on improving the OS for all ARM-compatible processors.
Figure 4: Windows Mobile 2003 gives us greater graphics performance.
Figure 5: As you can see, ActiveSync performance actually got worse with Pocket PC 2002, but we see huge performance gains with Windows Mobile 2003. Testing on the Pocket PC 2002 and 2003 devices was performed using ActiveSync 3.7 on a USB 2.0 port.
Figure 6: The platform index shows us that while Windows Mobile 2003 doesn't offer enormous enhancements over Pocket PC 2000, it does over Pocket PC 2002, which is where most of us are coming from.
Figure 7: Unfortunately, the file system is where the performance losses in Windows Mobile 2003 are most noticeable - this test measures how long it takes the Pocket PC to enumerate (list) a directory with 2000 files. It's not as grim as it looks though - this test is ranked very low in the Spb Benchmark scores, because while the chart would visually indicate a huge difference, it's really the difference between Windows Mobile 2003 taking 0.06 seconds to enumerate 2000 files, and Pocket PC 2002 taking 0.015 seconds. Both are fast. ;-)
Figure 8: Internal database speeds were one of the few things that got faster with Pocket PC 2002, and that trend continued with 2003.
Figure 9: The BitBlt graphics test measures how quickly the Pocket PC can draw an image to the screen using GDI functions. Windows Mobile 2003 shows a tremendous speed boost over previous operating systems, which impacts normal graphics operations like displaying pictures, simple games like Solitaire, etc. Games use GAPI, which this test does not measure.
Figure 10: A Windows Mobile 2003 device is capable of opening a 240 KB Pocket Word document nearly 300% faster than a Pocket PC 2000 device.
Figure 11: Yes, that's right - things got slower with Windows Mobile 2003 when it comes to browsing the Web. Pocket Internet Explorer is still a little pokey when it comes to processing HTML files. One of the reasons that Pocket PC 2000 is much faster than Windows Mobile 2003 in this test is that it's a much simpler browser and supports less HTML functions than Windows Mobile 2003. I feel there's a lot of room for improvement with Pocket Internet Explorer - Windows Mobile 2003 didn't meet my expectations in this regard.
Figure 12: JPEG loading performance is likely improved under 2003.
Figure 13: When it comes to opening a folder with 2000 empty files using File Explorer, Windows Mobile 2003 offers dramatic gains over Pocket PC 2002.
Figure 14: Although you'd think that ZIP compression speed would be solely a function of the CPU, Windows Mobile 2003 gives a solid 128% speed boost over Pocket PC 2002. Cool. 8)
Figure 15: ActiveSync speeds saw the greatest boost - I wonder if this means they finally got rid of the USB over serial driver that hobbled previous ActiveSync speeds? The speed increase in moving data from the desktop to the Pocket PC is so dramatic (139% faster) that this is the first generation of Pocket PC where I don't think people need to have an external memory card reader to load music onto their device - it's finally fast enough to do directly from desktop to Pocket PC. Well done Microsoft!
Figure 16: Although not as dramatic as the speed difference above, the speeds at moving data from the Pocket PC to the desktop PC are also enhanced.
Figure 17: This is the overall index showing how the three iPAQ 3650's each compare to a Dell Axim X5 running Pocket PC 2002, and a new iPAQ 2215 running Windows Mobile 2003. Incredibly, even the blazing iPAQ 2215 still trails the original iPAQ 3650 slightly. What kind of nitro did Compaq put in that device? I find it somewhat sad that, over three years after the fist iPAQ came out, we can't completely trump it with brand new devices.
Figure 18: I continue to have my doubts about how good of a CPU the XScale processor is. The performance dominance of the XScale PXA255 version is obvious when compared to the XScale PXA250 (Dell Axim), but when you consider that the iPAQ 3650 with a score of 1251 is running at 206 Mhz, the score of 1784 for the 400 Mhz XScale PXA255 becomes much less impressive. I hope Intel's next-generation XScale processor offers more impressive performance. I wish I had the Windows Mobile 2003 upgrade for my Dell Axim X5 so I could see exactly how much difference Windows Mobile 2003 makes on a PXA250. Was the PXA250 being held back by the Pocket PC 2002 operating system, or is it the other way around?
Figure 19: As readers of my iPAQ 2215 review will know, the 2215 benchmarked strangely slow in some graphics tests. This chart shows how poorly the 2215 benchmarks against the iPAQ 3650, but it also shows how much the OS alone improves the graphics performance, making the difference in performance of the 2003 iPAQ 3650 (1399) and the iPAQ 2215 (567) even more aappalling- a massive 247% difference! iPAQ 2215 owners take heart - it's been speculated that the Spb Benchmark tests don't tap into the MediaQ chip, so these results shouldn't be taken as a true measure of your device performance. Hopefully future versions of Spb Benchmark will take advantage of the MediaQ chip.
Figure 20: Further reiteration of what a big difference the operating system makes when it comes to ActiveSync speeds. The iPAQ 2215 pulls ahead here (as it should), and the speed of the Axim running 2002 shows that the more modern hardware can account for as much as a 52% speed boost, even with the same OS (2002 Axim vs. 2002 iPAQ).
A big thanks to Spb Software House for making such a powerful tool for us reviewers to use!
First, let's look at how the Windows Mobile 2003 iPAQ compares with the baseline index, which is based on a Pocket PC 2000 iPAQ 3650. Take a look:
• Overall, Windows Mobile 2003 is 15% faster
• The CPU performance of the StrongARM 206 Mhz is 25% faster under Windows Mobile 2003 than under Pocket PC 2000
• The file system is slower (0.3%), but only slightly so in the overall score (the new OS is much slower in some key tests you'll see later on in this article
• The graphics subsystem of Windows Mobile 2003 is vastly improved over previous versions - a whopping 40% faster.
• ActiveSync has the most dramatic improvements - an incredible 103% boost in speed over Pocket PC 2000.
Read on for the comparisons between Pocket PC 2000, 2002 and 2003 - I think you'll be surprised at how much of a speed drop we endured with Pocket PC 2002 in some tests, and what a dramatic difference there is in moving to the 2003 OS. Clearly, the resources required to move to the new Windows CE .Net 4.2 core were worth it, even if it meant that we didn't get that many new application-specific features in Windows Mobile 2003.
The screen shots below do not represent every possible test that Spb Benchmark is capable of performing, only the ones I felt were more relevant and showed the most dramatic performance differences. The indexes and averages are based on the entire suit of Spb Benchmark tests.
Figure 2: We lost speed with Pocket PC 2002, but gained it back (and then some) with Pocket PC 2003
Figure 3: Windows Mobile 2003 offers significant CPU enhancements for any ARM processor, without any special XScale tweaks. This is further proof to me that Microsoft made the right decision in not making special optimizations for the XScale processor, and instead focusing on improving the OS for all ARM-compatible processors.
Figure 4: Windows Mobile 2003 gives us greater graphics performance.
Figure 5: As you can see, ActiveSync performance actually got worse with Pocket PC 2002, but we see huge performance gains with Windows Mobile 2003. Testing on the Pocket PC 2002 and 2003 devices was performed using ActiveSync 3.7 on a USB 2.0 port.
Figure 6: The platform index shows us that while Windows Mobile 2003 doesn't offer enormous enhancements over Pocket PC 2000, it does over Pocket PC 2002, which is where most of us are coming from.
Figure 7: Unfortunately, the file system is where the performance losses in Windows Mobile 2003 are most noticeable - this test measures how long it takes the Pocket PC to enumerate (list) a directory with 2000 files. It's not as grim as it looks though - this test is ranked very low in the Spb Benchmark scores, because while the chart would visually indicate a huge difference, it's really the difference between Windows Mobile 2003 taking 0.06 seconds to enumerate 2000 files, and Pocket PC 2002 taking 0.015 seconds. Both are fast. ;-)
Figure 8: Internal database speeds were one of the few things that got faster with Pocket PC 2002, and that trend continued with 2003.
Figure 9: The BitBlt graphics test measures how quickly the Pocket PC can draw an image to the screen using GDI functions. Windows Mobile 2003 shows a tremendous speed boost over previous operating systems, which impacts normal graphics operations like displaying pictures, simple games like Solitaire, etc. Games use GAPI, which this test does not measure.
Figure 10: A Windows Mobile 2003 device is capable of opening a 240 KB Pocket Word document nearly 300% faster than a Pocket PC 2000 device.
Figure 11: Yes, that's right - things got slower with Windows Mobile 2003 when it comes to browsing the Web. Pocket Internet Explorer is still a little pokey when it comes to processing HTML files. One of the reasons that Pocket PC 2000 is much faster than Windows Mobile 2003 in this test is that it's a much simpler browser and supports less HTML functions than Windows Mobile 2003. I feel there's a lot of room for improvement with Pocket Internet Explorer - Windows Mobile 2003 didn't meet my expectations in this regard.
Figure 12: JPEG loading performance is likely improved under 2003.
Figure 13: When it comes to opening a folder with 2000 empty files using File Explorer, Windows Mobile 2003 offers dramatic gains over Pocket PC 2002.
Figure 14: Although you'd think that ZIP compression speed would be solely a function of the CPU, Windows Mobile 2003 gives a solid 128% speed boost over Pocket PC 2002. Cool. 8)
Figure 15: ActiveSync speeds saw the greatest boost - I wonder if this means they finally got rid of the USB over serial driver that hobbled previous ActiveSync speeds? The speed increase in moving data from the desktop to the Pocket PC is so dramatic (139% faster) that this is the first generation of Pocket PC where I don't think people need to have an external memory card reader to load music onto their device - it's finally fast enough to do directly from desktop to Pocket PC. Well done Microsoft!
Figure 16: Although not as dramatic as the speed difference above, the speeds at moving data from the Pocket PC to the desktop PC are also enhanced.
Figure 17: This is the overall index showing how the three iPAQ 3650's each compare to a Dell Axim X5 running Pocket PC 2002, and a new iPAQ 2215 running Windows Mobile 2003. Incredibly, even the blazing iPAQ 2215 still trails the original iPAQ 3650 slightly. What kind of nitro did Compaq put in that device? I find it somewhat sad that, over three years after the fist iPAQ came out, we can't completely trump it with brand new devices.
Figure 18: I continue to have my doubts about how good of a CPU the XScale processor is. The performance dominance of the XScale PXA255 version is obvious when compared to the XScale PXA250 (Dell Axim), but when you consider that the iPAQ 3650 with a score of 1251 is running at 206 Mhz, the score of 1784 for the 400 Mhz XScale PXA255 becomes much less impressive. I hope Intel's next-generation XScale processor offers more impressive performance. I wish I had the Windows Mobile 2003 upgrade for my Dell Axim X5 so I could see exactly how much difference Windows Mobile 2003 makes on a PXA250. Was the PXA250 being held back by the Pocket PC 2002 operating system, or is it the other way around?
Figure 19: As readers of my iPAQ 2215 review will know, the 2215 benchmarked strangely slow in some graphics tests. This chart shows how poorly the 2215 benchmarks against the iPAQ 3650, but it also shows how much the OS alone improves the graphics performance, making the difference in performance of the 2003 iPAQ 3650 (1399) and the iPAQ 2215 (567) even more aappalling- a massive 247% difference! iPAQ 2215 owners take heart - it's been speculated that the Spb Benchmark tests don't tap into the MediaQ chip, so these results shouldn't be taken as a true measure of your device performance. Hopefully future versions of Spb Benchmark will take advantage of the MediaQ chip.
Figure 20: Further reiteration of what a big difference the operating system makes when it comes to ActiveSync speeds. The iPAQ 2215 pulls ahead here (as it should), and the speed of the Axim running 2002 shows that the more modern hardware can account for as much as a 52% speed boost, even with the same OS (2002 Axim vs. 2002 iPAQ).
A big thanks to Spb Software House for making such a powerful tool for us reviewers to use!
