Iron Man wasn't even popular with the general audience.
Yep, the idea that only popular characters can be made into movies needs be thrown out of the window. Marvel proved that with Iron Man, Thor and arguably Captain America to a lesser degree. No one was particularly clamoring for those movies and now everyone loves them.
All it takes is a great actor, good script, writing and direction and you can most definitely have a hit with almost any major character or team within the X-Men universe.
There have been plenty of amazing solo runs from the X-Men - Gambit, Mystique, Deadpool, Dazzler, Magik, Longshot.
There are also plenty of other teams that are completely different from X-Men proper that can lead their own film (other than X-Force) - X-Factor, Alpha Flight, X-Corps, Excalibur, Hellions, New Mutants - they have plenty of options.
They shouldn't just limit themselves to what is popular. Any of these can be popular if done right.