Not a fan of that idea since it either means an all-white X-Men team with only one or two female members (if Polaris is included) or being unfaithful to the source material. I'd prefer if the X-Men kept Storm and Psylocke and hopefully also Bishop.
I'd do it like this. The current X-Men series continues in its own universe with Lauren Schuler Donner still producing but having to work under Kavin Feige with X-Men as its own brand so it doesn't cannibalize MCU release dates and unnecessarily prolong phases and make the wait times between sequels as long as it's been for Iron Man. Fantastic Four would join the MCU proper however and start off with an origin that's a combination of the 616 and Ultimate origins. The FF face off against Annihilus in a film that's described as, "Apollo 13 meets Alien" where if Annihilus leaves the space station and reaches Earth, all life on Earth will die. Reed finds a way to send him back to the Negative Zone without killing him and after the credits, he's found a way to open a wormhole and head back into our universe and appears in Skrull Space with a young Skrull boy pointing at the hole on the sky before we see the shot pan to Annihilus and his newfound ally Blastaar to set up Guardians 4. Doctor Doom would be the villain in Black Panther 2 and Kang would be the villain in Avengers 5.
X-Force would be assembled from across time and space with Cable, Deadpool, Domino, Psylocke, X-23, Rictor and a miraculously alive Archangel.
X-Men (with new members Gambit and Bishop) and X-Force would team up in a Messiah Complex crossover to defeat Mister Sinister while also fighting each other with Cyclops and Cable trying to save Hope Summers while Bishop and Wolverine try to kill her to stop the events of Logan from occurring. Gambit is torn between aiding the X-Men and Mister Sinister but eventually joins the side of good and saves everyone while Cyclops defeats Sinister once and for all (although the puddle of goop that he's reduced to starts moving after the credits).
House of M: The ultimate two-part Marvel film which would close off Phase 4 with a massive event. Doctor Doom tempts the Scarlet Witch with a world in which her brother is still alive. Wanda finds it too hard to resist and destroys both realities while merging then into one. We are now in a world where Magneto rules Earth, the Avengers are fighting to liberate the world from him despite Magneto taking over to prevent the humans from building Sentinels. Both sides have a point about the other supporting tyranny and the Fantastic Four are the only ones who realize that they're in an alternate universe since they were in the Negative Zone when universes merged. Doctor Doom eventually dethrones Magneto, forcing Reed to fight him and get reality restored by finding Wanda... who restores a reality where the Green Goblin is now the director of SHIELD.
Enter Phase 5: Dark Reign