Originally Posted by The Joker
Actually it was Rachel's death and the failure of his energy project that made him give up on life and become a recluse. He felt he had nothing left to do as Bruce Wayne ("There's nothing out there for me"). As Alfred reminded him, he used to talk about finishing being Batman and having a life beyond the cave, to which Bruce responded Rachel was his life beyond the cave.
Giving up Batman had nothing to do with it.
You're forgetting that Bruce was distraught over all the chaos Joker unleashed, leading to Rachel's death and Harvey's accident.
Its not so much just Rachel's death alone as it is what Rachel's death represents - Bruce's failure to protect Gotham's innocents, and the inadvertent rise of criminals due to Batman's presence.