I really doubt J-Law has that much appeal as an actress. The Hunger Games had massive appeal, so she pulled in a big fanbase because of that. But would all those Hunger Games fans rush to see her in X-Men? I doubt it. Especially when she moaned about the make-up and looked like she didn't want to be there.
She didn't bring any spark to the role at all. When a supermodel (Rebecca) can bring more edge and depth to a character than little miss Oscar-winner, you know something is up. When J-Law appeared on the Graham Norton chat show in the UK, they showed a clip from XM:A (where she talks to Xavier at the mansion) and she hated what she saw and said she was "a wildly bad actress." She just isn't into these films, she didn't even study acting and she seems to have a generally negative demeanour about everything. Good riddance. She's done nothing for this franchise to be honest.
As for McAvoy, he's just not leading man material. He's great but he just doesn't have megastar presence. That's no criticism of him, it's just the aura/status he has, it's where he fits in Hollywood's hierarchy. But that shouldn't matter anyway, because Xavier is not meant to lead the franchise - and the public will never accept such a non-action character as the headliner of a superhero series. Fox's tendency to try to make him front and centre doesn't work. So, McAvoy is fine for the role, but the tendency to make the role so central when the more dynamic characters are sidelined has not brought much success.
In the X-Men films, everyone is so moody and neurotic, battling with inner demons. It's just too much. Too many characters are haunted by some inner conflict and it lowers the tone of the movies into something quite glum and gloomy. Wolverine's struggle with his past, Jean's struggle with her powers, Cyclops' struggle with his powers, Xavier's struggle with whether he suppresses or accepts his students' powers, Magneto's struggle with his family tragedies. It's all a bit heavy for a superhero movie. At times we need a lighter touch.
Deadpool dealt with a character having cancer and then being experimented on but it didn't bring the whole movie down into a gloomy moody drama.