The X-Men movies were never as popular as Raimi's Spider-Man trilogy or The Avengers movies, but they still ranged from modest hits (the original movie, First Class) to massive successes (Days of Future Past, Deadpool 1 &2, Logan). I truly believe that at least four of the films in the franchise are among the best that the superhero genre has to offer. But critical praise and popularity aren't the same.
Wanna know a big reason why they weren't as popular as Spider-Man or The Avengers?
Merchandising. Or, lack thereof.
One of the reasons why the Spider-Man and MCU movies were/are so successful is because of their marketability towards kids. There is no shortage of Spider-Man, Iron Man, Black Panther, and Captain America toys, even if they're based on their comic incarnations and not necessarily MCU.
The X-Men movies though? It's a little bit more complicated. There was a line of action figures released to tie in to the first film back in 2000. They weren't bad, either. Almost like a precursor to Marvel Legends. They were a good mix of kid and collector friendly and there was even an electronic X-Jet and playsets to go with them. But after that, there was less and less merchandise for each sequel. X2 had a great line of action figures to go with it but there weren't a lot of characters available in it. The Last Stand had a line of comic-only figures to tie in with a few of the movie characters like Beast and Colossus showing up in some Marvel Legends waves. X-Men Origins Wolverine had a line of 4 inch figures (one of which made a cameo in Deadpool). And there wasn't much of a selection outside of figures either besides maybe some Wolverine play claws released during the first two movies.
After that, there was nothing. Everything from First Class onward had virtually no merchandise whatsoever save for a few items geared at adult collectors only from companies like Hot Toys. I believe there was something going on behind the scenes between Marvel and Fox over a merchandising dispute considering Fox would have received all profits from any X-Men movie merchandise being sold. At first this wasn't a problem since Marvel was selling its IPs to every studio under the sun, but once Disney acquired the company and Marvel Studios took off they naturally wouldn't have wanted Fox to see a dime of that money considering they were a direct competitor. Due to this, there was also a bit of a decrease in X-Men figures coming from Hasbro during this time compared to other Marvel characters like Spider-Man. It's a shame considering we're only just now seeing Marvel Legends versions of movie Deadpool, McAvoy Xavier and Fassbender Magneto now that Disney owns Fox, but better late than never I guess. Even so, those figures aren't aimed at kids anyway.
But I don't think kids really connected with the black leather clad X-Men of the films compared to the more colorful Spider-Man, Iron Man, or Captain America. Funny enough, I feel like I saw more kids in the screening of Deadpool 2 that I went to than any X-Men film I had seen in theaters since The Last Stand. To be honest, I don't think kids have really been overly excited about the X-Men since the days of the '90s cartoon. That's the only time in my life I remember there being more X-Men toys than Spider-Man toys on shelves.
Call me crazy but I think there's a real correlation there in terms of popularity among kids. They do make up a good number of the audience, after all.