Cap, with all his skills is still no match against even a bottom rank, still-afraid-of-yellow-sun Kryptonian, he was getting pwned by Loki single-handedly. Even moreso with Hawkeye, Widow, those arrows aren't gonna do ****. Iron Man needed a Supercharge just to stand toe-to-toe with Thor, and he's going to do what against Kryptonians with their own battle armor as well as superpowers? That only leaves Hulk & Thor to do the brunt of the heavy lifting and it wasn't a cakewalk for either of them against the Chitauri, glorified battle droids that they are.
The Chitauri are not physically enhanced, they wouldn't stand against Superman if he went full out, same for those leviathan ships. IT also wouldn't take much for Kal to take the portal device and the roof it was sitting on if necessary, and throw it back where it came from