I've dated multiple men outside of my race, and my current boyfriend is black, and I'm white. After 6 years, I can tell you that we have always and do still get weird looks out in public, depending on where we are. My family loves him, and I think that in your case, hopefully your family will be cool with it, because things are harder when your family isn't cool as opposed to strangers.
My ex before Spoons was also black, and my non-blood related redneck uncle whom I hated anyways threatened to shoot that "n word" with a shotgun in the front yard at Christmas one year. That was a horrible holiday, and I haven't seen or spoken to my aunt's husband ever since. Good riddance.
The main annoyance in general is how outsiders think it's their business to make nasty comments. I remember when Spoons and I were in college, one afternoon he told me how a black girl that we both mutually knew came up to him and asked him if he thought black women weren't good enough for him. **** like that pisses me off.
On a whole, however, depending on where you live, interracial dating is so common that I really doubt you'll have much to worry about. The random nasty looks are easy to ignore, and certainly don't come from everyone.