H.W. lost because of the economy. He had over 80% approval rating of his handling of the Gulf War and going into late-91 he was still in the high 60s. It was the economic discontent that allowed "Slick Willie" to get people's anxiety to put him in the WH. Otherwise, I doubt Clinton would have won.
As you said, look at 2004. While the Democrats were frothing at the mouth to get rid of George W. Bush who to them in '04 was "just like Hitler" and "worst president ever," etc. etc.
But while they were excited about Kerry, most of the country was rather ambivalent about him. The fact that Bush and Rove's campaign (as well as some sleazy 527 ads) was able to define Kerry as a lethargic, two-faced elitist who had no real ideas. If it is Romney, expect a similar response from this WH to the new Massachusetts pol who has flip flopped enough times to open a iHop (they just won't have to paint a war hero as a coward as the 527s did).
Bush wasn't exactly a national pride in 2004, but he was able to keep middle America unenthused about John Kerry and won reelection in a traditional result. I think Mitch Daniels is the toughest competitor for Obama right now. Romney, Palin, Gingrich or Barbour are not nearly a threatening (though the latter two could prove quite difficult if they campaigned well).
But again, if the economy is bad in 2012, Obama is in deep trouble, unless the Tea Party runs Sarah Palin through the primary like they did Sharron Angle and Christine O'Donnell this year. Carter and especially H.W. can prove that.
