I will concede every X-Men have characters who clutter the film only to fill in the background. But I wouldn't say the Avengers are immune to that (right, Hawkeye?).
I would also say that the main characters of at least the Singer and Vaughn X-Men movies are far better drawn or more nuanced than what Marvel has done. Intentionally so. The depth with which McAvoy and Fassbender's Xavier and Magneto are explored in FC and DOFP or Wolverine is in the first two Singer movies is much more thoughtful (not necessarily a good thing to audiences) than say Thor, Captain America, Black, Widow, Hawkeye, etc.
With that said, they also clutter every film with do-nothings like Angel (ether version), Storm (all films), even Blink, Bishop, and Colossus in DOFP.
So, it is a give and take. I think for the characters the X-franchise has zeroed on, there is a lot of depth there. But there are a lot of background fillers that Marvel avoids in the larger sense.