The Avengers and Inhumans connected during the Kree-Skrull War as well as at other times throughout their history. It would be easy enough to build on those connections from the comic book universe in the MCU. The key thing to remember is that the comics =/= MCU. Things will doubtless be changed, perhaps drastically, to fit the film universe, so concerns about connections between groups are without merit.
The idea that the GA would automatically associate them with the X-Men is laughable. It's not as if the mutants are the only characters with powers or strange appearances that the audience has seen. By the time an Inhumans movie comes out, Marvel will have shown the world the Frost Giants, the Chitauri, the Dark Elves, the Kree and other aliens, a gun-toting raccoon and a sentient tree. People will associate alien-looking beings with the Marvel brand rather than the X-Men. And of course, they'll see Marvel's The Inhumans
and know from the studio's name in the title what to expect.