Gambit could easily work with the OT cast film. They did bring Nightcrawler in X2 and Beast in X3 but just because Rogue is in the picture, it doesn't mean they have to do the Rogue/Gambit pairing.
Truth is they just screwed up that storyline by having Rogue in the OT cast and then having Gambit in Origins. Oh well.
In hindsight, the Origins film didn't any good to the series. They could have just moved forward with a X-Men 4 movie and introduced Gambit there.