We really have to think about this from a logistics point-of-view.
These actors, many of them at least, are a little older. Many of them have other things they plan to do with their careers. They aren't going to want to play the same character forever.
Not only that but eventually we, as fans, will get burned out on these films. General audiences will also, eventually, get burned out. That's just the sad inevitability.
I wouldn't be surprised if they are looking to do Avengers 2 and Avengers 3 back-to-back. Keep the proverbial momentum going. I'm with a lot of other people on these forums expecting to see A2 release in 2015, but I expect we will get A3 a little sooner than many expect, and it will drop in 2017.
I really think Guardians of the Galaxy is going to be Marvel's way of testing the waters on a non-Avengers movie. It may take place in the same universe, it may even have subtle tie-ins or references, but I don't think we will see Nick Fury or Tony Stark make a surprise after-credits appearance.