OK, I've seen the film a couple of times now, and I feel more prepared to answer. I am still a little unsure where to rank AoU, but I watched Movie Bob's review and his thought's seem to reflect mine when he talked about the comparison between Godfather and Godfather pt.2. The Godfather is a timeless classic and was a milestone in the movie industry, but The Godfather pt. 2 was the better story. That's how I feel here.
Avengers is a game changing classic film that will be remembered for all time. There are those certain films that people relate to life events, where they were, what they were doing, what was going on in the world. Avengers is that film for this newest generation.
Age of Ultron is the better story though, and a better representation of what the Avengers were in the comics. Avengers 1 there were all these influences of how Shield formed things under Fury's influence. Sure Fury is in this film and has a key moment, but this is all about the Avengers. Tony financing the team and using Avengers Tower as their base. Cap leading the team, with a very classic lineup (only minus Wasp and Ant-Man). This film just felt more like the Avengers I knew from the comics. The story was larger, with them globe trotting, not just stuck in New York.
Loki was a great choice for the first film, since that was the first enemy they faced in the comics, but Ultron is the Avengers' nemesis, and while they didn't have Hank Pym around, they handled the creation story flawlessly, and created a great memorable villain.
I love A1, and always will. I will never forget the feeling seeing Cap and Tony, my two all time favorite Avengers standing side by side for the first time. The chill I got down my spine seeing the team lineup in formation facing the Chitauri. But Age of Ultron is the better film, and receives my vote.
This!! ^

