That's part of why I like Cobie Smulders for Sif. She's like 5'9". Respectable warrior goddess size.
Also, people putting that pic of Sif next to Portman and saying she looks all right should probably be aware that that's Sif as whatever the Asgardian equivalent to a teenager is. Meaning she's like half a foot shorter than she'll grow up to be and her figure hasn't filled out completely. For comparison, here's what Thor (who's a little older than Sif) looks like in that period, by the same artist, no less:
Not a scrawny lad by any means, but certainly not as big as we'd like to see Thor on the screen.