I think the first two are clearly the worst. They are really lacking the whimsy and magic that drives the series. The cinematography and art direction makes it look like a theme park version of the books. Also, the "Monster of the Week" style storytelling wasn't as interesting, although that was an issue with the first two books as well. They just seem less important. If I rewatch the series these days, I often just skip the first two completely.
The others are all comparable in quality, but I'll draw special attention to the next three.
Prisoner of Azkaban is the one the re-invented the series and I really appreciate Cuaron for that. After the bland first two films, this one was a massive improvement in terms of aesthetic and the one that the following films copied. It isn't my favorite because I think Radcliffe didn't give his best performance here and I was never a big fan of the time travel stuff, but this is many ways is where Harry Potter became Harry Potter. It reminds me a lot of Dr. No in the James Bond series where it established the foundation, but was improved upon later.
Goblet of Fire is my personal favorite. That's the one which sets up the rest of the series with Voldemort's return and the first major defeat for the good guys. The tournament is a lot of fun and Moody is maybe my favorite teacher (besides Snape) in the series. It is also the one where I feel the leads start really transitioning from kids to teenagers and then adults. There is a real sense of growth and that the times are changing in this film. A lot of this is true in the book as well, and it is my personal favorite of the books too, so this film does have a big advantage in that regard.
Order of the Phoenix is interesting because unlike GoF which was my favorite of the books, this one was one of my least favorites. It was overly long and mostly felt like filler with Voldemort having a minor role after his big return in the previous book. It really felt un-needed and that the series could have easily transitioned from GoF directly to HBP. That said, I rather like the film adaptation which cut a lot of the fat from the bloated novel and did a great job bringing Umbridge to life as the major antagonist for most of the film. It isn't quite my favorite of the films, but it is one of the better ones and the one that I think was easily the biggest improvement from the book.
