I doubt they've completely disregarded TIH considering they made "The Consultant", which in itself ties up the lose end in TIH with Ross and Stark. But at this point I think it would be easier to start over yet again with the Hulk.
Story-wise it would probably be more realistic if the love interests were involved, but like people have said earlier, there wouldn't be enough time, and it wouldn't be that interesting for the main plot. The only characters I see being good/necessary additions to the Avengers, either through cameos or mentions, would be Peggy and Jane considering Cap is dealing with meeting the modern world, and being a fish out of water and Thors reason for returning was his girl. Betty and Pepper I don't see as essential, but like I said would be more realistic to include. For example Banner asking SHIELD to able to see Betty in terms for him helping them. And if Stark and Potts are still together, and Stark has moved to NY, I'd assume she'd be going with him.