Originally Posted by The Joker
When I look at the effect of the villains on Bruce in the trilogy, the one who had the worst effect on Bruce's life is the Joker.
Bane did break Batman's back and toss him in a pit for months. But the Joker killed his childhood best friend and woman he loved. He destroyed Harvey Dent (his ace in the hole), thereby forcing Batman to take the fall for Dent's crimes in order to save Gotham. These combined acts of evil turned Bruce into a miserable recluse for 8 years, who gave up on life, and made Batman into a hated, hunted figure.
The Joker's evil had a much greater and longer lasting effect and impact on both Bruce and Batman.
Agreed, Joker definitely affected him the most. He challenged his code, he took his great love, and he most of all made Batman turn on everything he believed in by using a lie to provide a temporary sham of a fix for his city instead of the true change he'd hoped to instill. Joker really did win after TDK.