Midtown Comics in Times Square is probably the biggest (2 floors), but its the busiest especially from Wednesday through the weekend, and you never know if you'll run into someone (I bumped into Dan Slott a few months ago during ASM#700) The Midtown Comics crosstown, across the street from Grand Central Station is more quiet, but at both the people who work at both are great. I havent been to the Wall street one though, so I cant tell you about that one.
Forbidden Planet around 14th street/Union Street is good too, they just moved down the street into a bigger building and completely remodeled, and they have a great selection of T-shirts (especially Street Fighter and Doctor Who shirts) and it's a 2 for $30 deal (from the last time I went there in June)
Jim Hanley's Universe just moved from across the empire state building into a new building as well, I think that it's now on 32nd street, but Im not sure of the exact area. They have a great selection of indies and Graphic Novels and collected trades and a good selection of shirts as well.