I think it totally depends on the character in question, and how far they stray from the original description.
Take Hugh Jackman as Wolverine for example. His face, hair, and athletic build were all perfect for the role. The only problem many fanboys had with the casting is Hugh's height being about a foot taller than the character's. But did it really make much of a difference in the film? Not really. And Tyler Mane as Sabretooth was so much bigger than him it wouldn't have made any difference in their fight scene anyway. I really couldn't see Tom Cruise in the role. He might be closer to the right height, but his look is all wrong.
Some people have a problem with Arnold Schwarzenegger as Conan The Barbarian. I, for one, loved him in that movie. It inspired me to read many of the novels and comics. The one thing that I think they should have done differently is he should have worn a black wig instead of brown extensions (or dyed his hair black and use matching extensions), but other than that I think he totally looked the part.
If someone is described as "big and muscular", I think it's a mistake to cast someone who's short and thin. And if someone is described as blond, brunette, redhead, etc, then the actor should either dye their hair or wear a wig. Basically, an effort should be made to get the look as close to the original description as humanly possible. If you make the character a little too tall or too short, I could care less. If you switch genders, race, hair colour, etc, then it starts to bother me.