I too think it will effect the chances of it. Say Hulk tanks, they can easily remove Hulk, but Captain America...? Ouch...
I have to question your lack of logic here. If Hulk tanked... why remove him from an Avengers movie?
My guess as to your thinking is this: "OMG, if Hulk was bad, people wouldn't want to go see him in another movie!"
That kind of thinking works if you are thinking about a Hulk sequel. But in a team movie, people want to see lots of heroes. Having Hulk in there doesn't hurt the film at all because the kickass heroes (like Iron Man) will be there. So if anything, people will be attracted to see the film because (1) it's a cross-over and (2) they get to see the kickass hero again along with other heroes helping him.
Even if Cap's film tanked, Avengers would still go forward. And if you read about the timing of the films, Cap and Avengers will be filmed around the same time. So if Cap fails, well... it would be too late for them to stop an Avengers film anyway.
Not only that, but the age of the actors and the amount of films they want to make is really hurting the chance to have iconic actors like Norton and Downey on screen together.
This makes no sense at all. Avengers will be released in 2011. That means it will finish shooting somewhere in 2010. Are you saying that in TWO YEARS, Norton and Downey will be... lol... too old to play their parts?
Two years is not a long time. I don't know why you're complaining as if it is.
Say they want a Hulk 2, Hulk 3, Iron Man 2, and Iron Man 3, as well as Captain America, Thor, and Ant man films.
No, Avengers is set to come before Hulk 2 and Iron Man 3.
So before Avengers, we'll see Thor and Iron Man 2 in 2010... Ant-Man (no date released yet)... and a Captain America movie that will come out a month or so before Avengers.
That's not a lot of movies at all.
By the time they actually get to the avengers it will be a freaking mess with some having aged to much / not wanting to continue with these characters.
Once again, you come up with this crazy "age" thing. In the span of 2 years, people don't age that much. Ian McKellan played Magneto in 3 movies that were spaced out over the course of 7 years. SEVEN YEARS!? Surely McKellan died before #3 was complete!!!!
Oh wait, Magneto looked fine in all 3 films. So I think Downey and Norton... over the course of TWO years... will look just peachy keen.

t:
As for "not wanting to continue", Norton will be contractually obligated to star in a second film, and Avengers is coming out before any Hulk sequel is planned (at least this is the plan so far). But in any case, Norton seems to enjoy the role and took an active part in creating it. I think it's safe to say he'd be willing to do one or two more.
And Downey was locked in for two IM movies, but has now said he'd be willing to do many more because he had fun making the first.
As much as I want the Avengers, I have no idea how it will play out.
It will probably play out as a movie where all the heroes fight together.

t:
No time will need to be wasted on establishing each hero's origin... that's what all these solo films are doing. Hell, the solo films are even including after-the-credits scenes that let people know "these heroes might be working together in the future".
So all Avengers has to do is have a reason for all the heroes to fight together. So the film will need to create a common enemy for the heroes, and then the heroes team-up to take the threat out. Add in some great special FX and dialogue, and the movie will do fine.