I think the Ledger Joker is better and I guess I could even say I like him more than Nicholson (just like I prefer the Keaton Batman to Bale), but I still love the Nicholson Joker and think he did a terrific job. There's nothing I don't like about him in the film. I love his origin, I love his end game, I enjoy everything about him. Every time he's on it doesn't get old, same with Ledger.
Ledger's take on the Joker is sort of immortal in a way. The fantastic hype (that delivered, unlike most comic book movies these days), his unfortunate death, not being able to shed light on his actual performance through post-film interviews/Nolan and WB. being mum with behind the scenes material, etc. The character even lives on at the end of the film only to never appear again. There's almost this mystical quality to it PLUS the actual performance itself with all it's great character, quirks and such. It's great.
I don't like to compare them, because they both did the character justice, just in different ways. What one has, the other doesn't it, which isn't just true of the Jokers, but of both Batmans, and the whole, "Burton vs. Nolan" thing as well.
To put it this way, I don't think Nicholson or Ledger did anything "wrong" with their performances. Out of all the cinematic comic book villains from DC, Warner Bros, Marvel, Disney, Fox, Sony, etc. I rank Ledger and Nicholson at the very top.
As for "scariness", I think the Ledger Joker is a lot tamer than what I imagined before July 2008 where we were getting those "reports" that the character picked his scars, wore make up that started to rot, etc. The Brian Azzarello Joker is what I would consider "scary" and just downright ruthless and disgusting. I mean, he skins a man! He shoots a guy through the neck! He rapes his girl!
But the Ledger Joker not being that extreme isn't a bad thing. He wasn't camp by any means, but I never felt "afraid" of him. He was more fun than truly evil. He had a different kind of edge to him and was still a colorful, likeable villain even with some of the nasty things he did. I laughed at the mob meeting with Ledger's little idiosyncrasies like replying to Gambol with "yeah", or his child like behavior with Rachel or even how playful he was at the MCU. There's that great moment where that goon is complaining that his stomach hurts and it cuts back to the Joker looking the other way like, "oh crap, I don't know what he's talking about". I thought him being disgusted at the look of Dent's injuries, "forgetting" Rachel's name, and using handsanitizer was fantastic. It's great.
The only time he's freighting, to me, is the Why So Serious scene with Gambol, that's it. But him not being scary isn't a bad thing or negative thing. Quite the contrary.
Nicholson's the same way. I don't really think he's all that scary either. Like Ledger, he has moments or scenes where's he looks scary, like the post-surgery scene with the mob doctor or when he kills Grissom, or just his look overall with the injuries he sustained (especially at the mob meeting with the flesh make up and brimmed mob hat) but I was never frightened. The thing with Nicholson's Joker was how disturbing he was. I could buy that he was someone with "bad wiring" before the Axis Chemicals incident and falling into the vat triggered the Joker persona.
I look at the two this way. The Ledger Joker wanted to prove that everyone's spirit was as ugly as he was while the Nicholson Joker wanted to prove that everyone's vanity was as ugly as he was.