I think you guys really hit the nail on the head. In a way, The Avengers served as a double-edged sword. Yes, we got a fantastic movie out of the deal, but in a way fans now expect all their favorite characters to be in the next Avengers picture.
It's not only unrealistic, but it would be detrimental to the film as a whole to cram that many characters into one movie.
Dr. Strange IS a complex character that certainly deserves his own movie. I've said it before: I really think GotG is going to be Marvel's way of testing the waters on a non-Avengers flick. They may have a connection through Thanos, but I wouldn't want both teams appearing in the same movie. It's just too much and no character would be done justice in a movie like that unless the running time was somewhere around five to seven hours.
I'd much rather have a fantastic set of movies than have a sub-par movie that manages to get all my favorite characters in. If you want a bunch of half-developed characters running around and blowing stuff up go rent G.I. Joe.
Yes, the MCU should be interconnected, but that doesn't/shouldn't mean that all those characters should be in the same battle or even the same movie.
To give an example from a completely different genre and medium: In the early 90's there were three sit-coms on NBC that took place in the same universe: Mad About You, Seinfeld, and Friends. They established this by interconnecting Friends and Mad About You through a minor character who happened to appear in both shows. They connected Seinfeld and Mad About You through having a main character from one meet a main character from the other in one episode. That was it.
There's really no reason for Dr. Strange to be in The Avengers at all beyond, maybe, a mention or a cameo. Save that spot on the roster for characters that were integral to The Avengers and vice versa, i.e. Ant-Man and Wasp.