I can add some more fuel to that fire:  I read an interview with him just as X2 was being released, where he said he regarded movies like that as "bank" movies--where he could make a lot of money, put it away, and move on to smaller projects where he wouldn't have to worry about the money as much.
I have read that he hated the makeup, I've also read that he possibly didn't get along with Bryan Singer, but that's all gossip, so who knows what's true.
And I can't diss the guy....I saw him on Broadway in "Cabaret" a few years before X2, and he was amazing.  My friends waited for his autograph after the show, and he was really nice.  Plus I saw him at a CD signing for BC/EFA where he did a Q&A, and again...he was really nice.