It's far from dumb action.
The movie has more heart then most other superhero movies, and easily surpasses the Spider-Man films in terms of the reverence and love of the characters.
Cap and Iron Man have an interesting relationship, and its very antagonistic.
Thor and Hulk are like drinking buddies.
Black Widow is the lone woman out, but takes comfort in her non-romantic partnership with Hawkeye, who is the one who was key to her redemption from villain to hero.
But the best relationship is Tony and Banner, with Tony being the person who truly believes and is not afraid of Banner. Steve isn't either, and tells Banner upon introduction that the only thing he cares about is that Banner is a brilliant mind -- could care less about the Hulk stuff. It's a quick, great moment. And another great dynamic to the relationships is Banners interaction with Widow, as she has a great emotional/terrified moment with him that rises to the top of my favorite moments in the film.
But Tony and Banner have a wonderful interaction that builds to one of the GREAT moments of the film in terms of Hulk.
The action is completely integral to the plot, and arises out of the team coming together and making a decision that goes beyond even Fury. And they do it as a team, realizing very subtly their responsibilities to one another and the world.
By the end, these guys are friends and you believe it and the way they found one another is active and action-packed, so the action always seems like currency for the larger relationships at stake.
It is this years Star Trek (2009) in that way, where the action services the script and the stories at all times.