I'll be honest, I think The Avengers is kind of a thankless job for a director. You're kind of tied in to the visual worlds that others have created, significant parts of casting have been taken out of your hands, and you probably have limits what you can do storywise with most of the characters. Having to work within other people's continuity and characterization, doesn't sound like a lot of fun. Or creatively fulfilling. You can play around with the plot some, and the character interactions, but it's a fairly limiting project.
So, it's no wonder that a proven big name isn't really interested. And, heck, Vaughn sounds like he'd rather pursue something like Sandman than the logistical nightmare that The Avengers is likely to be.
In a certain sense, this is the least amount of control Whedon has had in probably over a decade. Yeah, it will probably be a hit regardless of how it turns out, but the odds are that it will be pretty messy and Whedon will have a tough time claiming credit for the success of the film. Nor is it likely to make it easier for Whedon to get his original projects made.