Just like Nolan? After TDK came out, his response to questions about a sequel was something to the effect of, "Do 3rd installments ever turn out well? Yeah, I thought so. There's your answer."
Whedon's also talking about what a huge, exhausting undertaking the first Avengers was and how he also has other things he wants to do. It's typical director speak to deny plans until they've done their contract negotiations and signed on the dotted line. Doesn't mean they don't actually have plans.
Now, for the record, I'm not saying Webb is definitely returning. I personally think that could go either way at this point, especially with him still having a standing contract at Fox. I'm just saying his silence on the issue is in no way proof that he's not returning.