I think you are overstressing the brutish elements, just as others are over stressing the gentleman aspect. Bond was never really "brutish", that is a bit of revisionist history from people trying to justify Craig's less orthodox looks. Bond has always been a gentleman adventurer in the classic British imperialist model, hence all of the comparisons to Saint George in the books. But gentleman in the Fleming sense is not necessarily one of manners, but one of station and social class. Bond is described as handsome throughout the books. When Vesper compares Bond to Hoagy Carmichael (who was a heartthrob singer), it is in the sense that he is good looking and manly like Carmichael, but she differentiates him on his coldness of personality and bearing. The cruel mouth and cold grey eyes. Bond is also a little darker of skin than your typical white Englishman of the 1950s/60s given the tan Bond picked up on his adventures.
Dalton was pretty much perfect in terms of having the look that Fleming described. While not as close to the book description in terms of physical details, Connery had that exact same blend of rugged attractiveness and cold, detached personality.
They have aged out of the role now, but the likes of Michael Fassbender, Aidan Turner, and others in that mold would have been perfect 5-10 years ago.