Looking at this mostly through the sequels, even though Age of Ultron doesn't actively annoy me the way that Guardians Vol. 2 does, I have to go with Gunn. His style, voice, and writing are consistent between both films, even if I think he overdoes it in the sequel. The same can be said for Whedon, but the first Avengers is very much in line with his style and writing.
He had a much bigger task to deal with coming in, with Avengers being only his second film after Serenity, but Age of Ultron does reek of meddling and overstuffing in a way that Guardians Vol. 2 doesn't. As a result, Whedon's style, I feel, often gets lost in favor of servicing future films, rather than servicing Age of Ultron itself.
Again, as much as I dislike Guardians Vol. 2, I can safely say that is James Gunn's film, but Age of Ultron doesn't have as much of Whedon's flavor as the first Avengers film does, so I give it to Gunn.