The dominance of the iPod didn't help either, as by this point it had rendered other portable music players all but obsolete.
It was only with the release of the Sony Ericsson W800i in 2005 that phones really started to look like they might be able to compete in the music space. The W800i had several things going for it but arguably the single biggest key to its success was its Walkman branding.
After the iPod, Sony's Walkman was probably the most recognisable MP3 player brand and the Sony Ericsson W800i was the first phone to carry that moniker, giving it a built in fan base.
The W800i had more than just strong branding though, it also had a button on the front with the sole purpose of launching the phone's MP3 player, with another on the side for pausing or playing tracks. This put music front and centre and made it easy to access and control.
That's a key feature as it kept things simple. Indeed as today's phones get ever more complex, Luc Burson, senior sound designer at HTC, believes that usability and simplification are major areas of improvement needed for smartphone audio players: "The biggest opportunity for innovative ideas right now is usability - all parameters of usability."
"Some of us can remember when modular music systems were all there was in the home and how easy it was in terms of usability and simplicity. There is major opportunity in smartphone audio experience for simplification. Retaining current abilities and integrating new forms of A/V I/O is key, but simplicity is paramount as a front-end."
On top of proving user friendly, the W800i didn't have the storage problems of earlier phones either as while it only had 34MB of onboard memory it also had a Memory Stick Pro Duo slot with support for sticks of up to 2GB. At 900mAh the battery was also a reasonable size, allowing for up to 30 hours of music playback.
Apple iPhone
By 2007 Apple's iPod range had been the biggest force in portable music for some time and it was only with another Apple release, the iPhone, that this looked likely to change.
The original iPhone was a perfect storm of recognisable branding and iTunes support, allowing millions of iTunes users to play their music on their phones and even to download songs straight to their iPhones from iTunes. That in itself was a massive deal as iTunes was the first and last stop for digital music in many people's minds.
The iPhone also had a good audio chip (the Wolfson WM8758BG), which supported 3D audio effects and gave it superior sound quality to many earlier phones.
Yong-hyuk Na describes the audio chip as the "heart of the internal hardware that delivers sound to the user...responsible for managing, converting and handling various sound sources such as the speaker, mic and headphones", so it's of vital importance, as without an audio chip, no matter how good the speakers or other audio hardware is it won't function.
The quality of audio chips has levelled out some now, according to Yong-hyuk Na, who told us that "performance among major audio chipmakers has standardized to a point where differences in audio quality can't be clearly distinguished, but can be realised through the optimized design and sound tuning capabilities of smartphone manufacturers."
So at this point placing too much importance on the chip itself would be wrong. But back in the days of the original iPhone it was a major factor.
Add to that support for gapless playback and a player layout which was a lot like the one on iPod's, making it instantly recognisable and intuitive and it's no surprise that the iPhone was such a hit with music fans.