The modifications vary massively - some just provide you with stock Android, whereas others completely change the look and feel of the phone. Mostly, the ROMs are developed by teams of hobbysists in their spare time, in return for no (or very little) money.
Installing a ROM is just a little more complicated than rooting. You have to install a custom recovery – a piece of software separate from Android, kind of like a very limited alternate operating system.
To start with. you'll need to boot the phone into 'recovery mode', usually achieved by holding some Konami-code-esque combination of buttons as you boot the phone up.
Once in the recovery, you can make system-wide changes like installing ROMs, deleting user data and more. The stock recovery that ships with phones is often very limited, so one of the steps in rooting a phone is normally to install a custom recovery.
There are two particularly common custom recoveries, ClockworkMod and TWRP. Just like the custom ROMs, these are freely available bits of software made by hobbyist hackers, not the actual phone makers, and can be every bit as buggy as you'd expect.
So, once the recovery is installed, all you have to do is find a custom ROM that'll run on your phone, download the file off the internet, copy it onto your phone, then boot into the recovery and flash away ('flashing' basically meaning installing).
Finding a good ROM is probably the most difficult part of the entire operation. One of the most common mistakes for first-time modders is downloading a ROM intended for a slightly different model of smartphone.
Owing to the different mobile networks in various countries, there's normally a couple of versions of any one smartphone in circulation at any one time, which are normally visually identical but sufficiently different on the inside, so that trying to install the wrong bit of software will screw them up.
Mainstream ROMs
That said, there is at least a fair bit of choice. There are a number of 'mainstream' custom ROMs, like CyanogenMod, Android Open Kang Project, and the British attempt, MoDaCo. Some focus on upping the processor speed, whilst others offer a vastly different user interface, or better battery life.
If you have a recent, mainstream flagship phone, you'll most likely have all the choice you want; older or less well-known phones don't get the same selection of ROMs, as the demand isn't quite there.
Compounding the problems is the bugginess of many ROMs. Remember, these are bits of software written by enthusiasts in their spare time, and tested by internet forums – not quite the same quality control that big manufacturers go through.
As a result, the latest releases of software can be buggy, and probably aren't recommended to install. At the very least, make sure you read through forum threads to see what problems others are having before installing – problems like the camera or touchscreen stopping working aren't unheard of.
If you do screw your handset up when trying to root or flash a custom ROM, chances are it's easily reversible. Because most features relating to uninstalling or reinstalling Android can be controlled via a PC and a microUSB cable, most problems can be solved by plugging your phone back into your computer, and re-flashing the stock software.
As a general rule, if you can turn your phone on and get into recovery mode, but not the main operating system, it's 'soft bricked'; this is opposed to 'hard bricked', which is when you can't turn the phone on, and it won't talk to a PC. At that point, your options are pretty limited, beyond sending the phone back to the manufacturer and hoping they don't ask too many questions.
Rooting and modding handsets isn't for the faint-hearted – it'll take at least an hour or two to get set up initially, and there's always the outside possibility of bricking your phone – but in many cases, the rewards are worth it.
Whether there's a particular feature that drives you crazy on stock software, or you're tired of waiting half a year for an Android update, most every problem can be solved with a bit of modding. Just don't expect everything to work perfectly, all the time.