Yeah, I remember hearing some former SEAL give an interview on CNN where he said that the hardest part was trying to have a family since you can literally be called for duty at any time of the day, and you would have to leave your family for long months with no contact and stuff.
That's why I have nothing but respect for anyone in the armed forces, because its really a tough job on all levels (physically, emotionally, and psychological), not to mention you're always putting your life on the line. I know I'm not strong enough to do that, but I commend those who are.