I dont think that X3 and Wolverine had nearly the negative impact people want to believe.
Comic book fans may not have liked them, but I have yet to find anyone in the general audience that didn't. I've had numerous people tell me how awesome they thought wolverine was, and how they thought it was the best one, and react in total shock when I tell them the fanboys didn't like it.
You can say its a small sample size or whatever, but I feel its pretty telling when its a consistent response across the board. Even today I had a friend give me a look of "wtf??!" when I told him a lot of people didn't like wolverine.
I think its probably more the fact that the franchise is starting to over extend itself. There's the fact that its a June release when all the other movies since the first one have been May releases. There's a lot of factors, and not saying the other movies aren't a factor, but to put them on the same level as a batman & Robin sized obstacle I believe is misguided.