I remember those people, along with the outcry after Ledger was cast in TDK. Let's face facts, Jack's Joker is still pretty awesome, but at the end of the day it's still Jack playing Jack.
Ledger's casting is probably the only time I remember there being some divisiveness upon a casting announcement which turned into unanimous praise once they saw him in the role. I thought he was a left field choice when he was cast but I trusted Nolan since he hit it out of the ballpark with Batman Begins. I kept an open mind but I was sold the instant I saw him in the first theatrical trailer for the movie. Even people who didn't like the movie still praised Ledger's Joker. Without him, The Dark Knight still would have been successful, but I don't think it would have been the juggernaut that it became.