Of course not. We "naysayers" are just saying that Marvel/Paramount's job would be easier if IM2 had been TF2-big instead of not being as big as the original. IM1 was a pleasant surprise, but its box office wasn't mind-blowing. It still had room to grow, as TF2 showed. I mean, it was a movie about a bunch of cars that turn into robots.

Although, IM1's box office may turn out to seem more more impressive over time, depending on how Thor, Cap, and Avengers do. (See below example re: Watchmen.)
It has nothing to do with "quality," since quality is subjective. We're talking about hype and momentum.
And considering how poorly Kick-Ass did this year, maybe the expectations on Watchmen WERE completely overblown to begin with. And Kick-Ass was marketed more fun than Watchmen certainly was.
We'll see if this is the case with IM1 and 2 a few years into the future. Right now, with the current movies available to us to compare it with, (especially TF2) people are just saying they thought it would do better than it's doing. Almost nobody (official peeps, people on BOM and WoKJ) predicted that IM2 would earn less than IM1, but it's looking very very likely.