While I personally enjoy BB as much as TDK, it's not hard to understand why the movie never became a mega-giga-hit (though it was still successful, bear that in mind).
1) like it was already said, the last memory of a Batman movie that people had was of B&R. Which, in case that anyone doesn't know, was so bad that it killed the previous series and everything had to be rebooted 8 years later.
2) there was a 8 year gap between the last Batman movie and BB (and the said last Batman movie was awful, to boot). That's a long time and people actually may have lost interest in Batman. During that 8 year gap a lot of other superheroes went to the big screen (Spider-Man, the X-Men, Blade, the Hulk, the Punisher, Daredevil) and that may have shifted the public's interest away from Batman, even though most of their movies didn't do very well.
3) BB wasn't marketed very well.
4) the movie itself, while (very) enjoyable to me - and I assume to most Batman and comic book fans - isn't the type of film that would attract audiences to see it again and again and again. In fact, every non-comic book fan that I know, including some of my best friends, thought that it was dull and boring. For the life of me, I would never understand why they thought so, but it is a fact. It's a movie with a lot of talking and little action, which is not the best recipe for a superhero flick; TDK had much more action in it, for example.