I'm not worried about Ruffalo being able to play Banner, or the Hulk for that matter. I think the hang up here I'm sensing is the look of Mark Ruffalo, whether people believe he can take on the guise of Banner. I certainly don't think Banner when I see Ruffalo, but I've tried to put this in perspective. Banner is essentially a skinny white dude with glasses and brown hair (sometimes sans glasses), and it's rather hard to be overly critical of Ruffalo without having seen him in any Marvel movie screen shots. About the only actor I've seen who looks like Banner is Guy Pierce, who looks exactly like Byrne's Banner. Other than that you have Bill Bixby, who did great, and Eric Bana who wasn't bad, and Edward Norton, who also did a good job. None of them look particularly like Banner, none of them have a history of playing scientists, and many of them had very un-Banner like roles beforehand. Ruffalo could be like Heath Ledger. Heath Ledger was doubted from day one, and look at how that turned out (I mean except for the whole 'him dying' part).
Ruffalo fills all the basics for the moment. He's white, not really muscular, and he has brown hair. Bingo-bango-jango. If he shows up looking like he did on the set of Collateral I'll be worried.