I love the scene in the cafe, when the guy from a old job starts talking to him, and he's just like "How bout this..." Awesome.
I think the film consciously shifts in the second half though. It becomes a bit more crazy and violent as the Driver does. I think from that moment where Irene tells him her husband is coming out of jail, that's it for the Driver, who is clearly socially awkward and I think he sees Irene and her kid as some kind of hope to a normal life. And he sacrifices himself, in that he does horrible, violent things, to protect them. Sorta like how Batman does it at the end of TDK.