Tuesday, May 20, 2003
Palm Commissions Competitive Analysis Report
Posted by Ed Hansberry in "THE COMPETITION" @ 12:30 PM
Report Available here in PDF format
Palm commissioned VeriTest to compare the Tungsten T and Tungsten C with the HP iPAQ 1910 and 5450 Pocket PCs. They looked at battery life, document size and WiFi performance. Of course Palm won hands down. No one commissions and releases a report when they lose. ;)
Palm developed the testing methodology with VeriTest to encompass real world usage. Just for reference, remember Palm defines real world usage in terms that allows them to say their devices have 3 weeks of battery life. Palm also paid for all of the devices. All were off of the shelf items except for the 'C which was unreleased.
The highlights of the test are:
• The Tungsten C lasted 1hr 35min longer than the iPAQ 5450 in the 802.11b test. Screen brightness was set to 50% on both devices and screen autodiming was disabled on the 5450. Question: Is it a real world test to disable autodimming on the iPAQ? Just because the Tungsten C doesn't support autodimming doesn't mean it isn't part of the iPAQs battery features.
• The Tungsten C was more than twice as fast as the 5450 in WiFi downloads. This test is misnamed. It was a HTML rendering test because they timed the download and rendering of CNN.COM, a 177KB page. Finally! Conclusive proof PIE's rendering speeds leave a LOT to be desired. The question is, what was visible in the browser's window when done? According to an MSNBC report the Tungsten C lacks the ability to render complex pages at all. I had to reset the Tungsten C before it found my home WiFi network. Once it did, it locked in for a smooth ride. Surfing on the Web via PalmSource’s Web Browser was very, very speedy for a Palm device. But it flunked my first test: drawing MSNBC.com’s home page. It didn't even come close: One part of the page was screen left and the bulk of the page was eight or nine screen widths over to the right. The Opera browser on the Sharp PDA, Handspring’s Blazer, and Pocket PC’s IE browsers all do a much better job of handling intricate Web pages. For now, I'll take readability vs. speed right now. Still, Microsoft, we need a better rendering engine than what PIE 3.0 currently has.
• The Tungsten T outlasted the iPAQ 5450 by a factor of five in the leave-the-device-in-your-desk-for-a-month test. The T|T lasted 21 days, while the 5450 lasted only 4 days. I think this was done by running the device down until it went into a power saving mode. Then they waited to see how long it would take before you applied power and the device woke up to a hard reset. I don't know if they configured the 5450's data conservation settings or not. I honestly don't know if the 5450 has that setting - I know the 3900 does so I am assuming the later 5450 does. Regardless, the 5450 wouldn't come close to 21 days in any event.
There were several speed tests shown, some of which were not quite on an even scale. First, they compared the video speeds of the Tungstens to the iPAQs. The 1910 doesn't have the latest X-Scale tweaked player and neither the iPAQ nor 1910 are tweaked much (if at all) for the ARMv5 processor, which the X-Scale is. Furthermore, both have the old PXA-250 processor that Intel has as much admitted isn't that good at processing code developed for the ARMv4 chips like the StrongARM. Palm OS5 devices were designed from day one as ARMv5 devices, so have a bit of an advantage. Doesn't invalidate the results one whit. Palm waited until the fall of 2002 to come out with a powerful OS and of course chose the latest processor technology. Microsoft developed the core of Windows CE in 1997 and with legacy devices and code to support, they can't drop everything and make a new ARMv5 only class of product.
The document storage thing is laughable to say the least. I won't touch on it but to say that my iPAQ currently has over 550MB of total storage capacity between the 512MB SD Card, the file store and RAM allocated for storage. I'm not concerned that my Excel spreadsheet is 261K rather than 163K on the device. At least I can open that spreadsheet from an email without having to rely on a desktop converter. ;)
I think it is an interesting study, but some of it should be taken with a grain of salt. The settings they used, like 50% volume and 50% screen brightness, are meaningless. They should set each device so they are putting out the same amount of decibels and lumens. Still, Palm has come a long way in the past few years. As Palm's own Chris Dunphy put it recently, PalmOS has been busy evolving out of the dark ages. I agree. This test should keep Microsoft on its toes.
Palm commissioned VeriTest to compare the Tungsten T and Tungsten C with the HP iPAQ 1910 and 5450 Pocket PCs. They looked at battery life, document size and WiFi performance. Of course Palm won hands down. No one commissions and releases a report when they lose. ;)
Palm developed the testing methodology with VeriTest to encompass real world usage. Just for reference, remember Palm defines real world usage in terms that allows them to say their devices have 3 weeks of battery life. Palm also paid for all of the devices. All were off of the shelf items except for the 'C which was unreleased.
The highlights of the test are:
• The Tungsten C lasted 1hr 35min longer than the iPAQ 5450 in the 802.11b test. Screen brightness was set to 50% on both devices and screen autodiming was disabled on the 5450. Question: Is it a real world test to disable autodimming on the iPAQ? Just because the Tungsten C doesn't support autodimming doesn't mean it isn't part of the iPAQs battery features.
• The Tungsten C was more than twice as fast as the 5450 in WiFi downloads. This test is misnamed. It was a HTML rendering test because they timed the download and rendering of CNN.COM, a 177KB page. Finally! Conclusive proof PIE's rendering speeds leave a LOT to be desired. The question is, what was visible in the browser's window when done? According to an MSNBC report the Tungsten C lacks the ability to render complex pages at all. I had to reset the Tungsten C before it found my home WiFi network. Once it did, it locked in for a smooth ride. Surfing on the Web via PalmSource’s Web Browser was very, very speedy for a Palm device. But it flunked my first test: drawing MSNBC.com’s home page. It didn't even come close: One part of the page was screen left and the bulk of the page was eight or nine screen widths over to the right. The Opera browser on the Sharp PDA, Handspring’s Blazer, and Pocket PC’s IE browsers all do a much better job of handling intricate Web pages. For now, I'll take readability vs. speed right now. Still, Microsoft, we need a better rendering engine than what PIE 3.0 currently has.
• The Tungsten T outlasted the iPAQ 5450 by a factor of five in the leave-the-device-in-your-desk-for-a-month test. The T|T lasted 21 days, while the 5450 lasted only 4 days. I think this was done by running the device down until it went into a power saving mode. Then they waited to see how long it would take before you applied power and the device woke up to a hard reset. I don't know if they configured the 5450's data conservation settings or not. I honestly don't know if the 5450 has that setting - I know the 3900 does so I am assuming the later 5450 does. Regardless, the 5450 wouldn't come close to 21 days in any event.
There were several speed tests shown, some of which were not quite on an even scale. First, they compared the video speeds of the Tungstens to the iPAQs. The 1910 doesn't have the latest X-Scale tweaked player and neither the iPAQ nor 1910 are tweaked much (if at all) for the ARMv5 processor, which the X-Scale is. Furthermore, both have the old PXA-250 processor that Intel has as much admitted isn't that good at processing code developed for the ARMv4 chips like the StrongARM. Palm OS5 devices were designed from day one as ARMv5 devices, so have a bit of an advantage. Doesn't invalidate the results one whit. Palm waited until the fall of 2002 to come out with a powerful OS and of course chose the latest processor technology. Microsoft developed the core of Windows CE in 1997 and with legacy devices and code to support, they can't drop everything and make a new ARMv5 only class of product.
The document storage thing is laughable to say the least. I won't touch on it but to say that my iPAQ currently has over 550MB of total storage capacity between the 512MB SD Card, the file store and RAM allocated for storage. I'm not concerned that my Excel spreadsheet is 261K rather than 163K on the device. At least I can open that spreadsheet from an email without having to rely on a desktop converter. ;)
I think it is an interesting study, but some of it should be taken with a grain of salt. The settings they used, like 50% volume and 50% screen brightness, are meaningless. They should set each device so they are putting out the same amount of decibels and lumens. Still, Palm has come a long way in the past few years. As Palm's own Chris Dunphy put it recently, PalmOS has been busy evolving out of the dark ages. I agree. This test should keep Microsoft on its toes.
