Frankly, I'm just tired of the 'oh woe is me I have dreams' We've known that for 8 seasons. I think it would be more effective for him to just put up with it, Quit with the damn memories and the rhetoric of ' i had good thing goin when you're gone blah blah' And at the end of his current quest. Quit. Period. Done. And that is that. Lol. That's just me venting a bit, cause really other than when he didn't have a soul, it's been the same old song and dance since this series began, he's just spewing it verbally even more than he used to.
As far as his memories of the good times, Maybe he just left because he has an unwavering loyalty to his brother (though by the end of this season I bet it will be gone) and has to finally finish the things he started. Sure, he left in the middle of the night, but I think that is just so he doesn't have to explain, and the memories he had actually were that good and they're worth remembering in a life of demons, monsters and killing