I can't see CBS giving it to Ferguson. He just isn't a well known commodity. He works for the more niche 12:30 slot, but not so much 11:30. Although, one nuance is, if CBS wants to keep the Late Show brand, they have to keep working with Letterman's production company (as I believe they own the show), so Letterman will get at least some say in his successor. Ferguson's show is also produced by Letterman and I believe the two have a decent work relationship.
I could see Conan being a possibility, though I don't know what his contract with TBS is like.
I personally don't like the idea of giving it to a woman simply for the sake of doing so. Especially when I can think of just about no one who is a good fit (no, Ellen is not a good fit outside of day time TV....these are two very different beasts). I think Conan is probably the best choice, but I'm not sure that he'd leave TBS.