Brubaker's X-Men run will always be tainted to be due to his pointless killing of Banshee. Not to mention he followed the only formula he knows, changing the past (Vulcan's group of X-Men) bringing back dead people (D'Ken), and killing people off (Banshee, Corsair, D'Ken). And he created and played up Darwin for the story when he could have used Lifeguard, who fit the story perfectly.
As for between Seagle and Brubaker, I enjoyed Casey's run quite a bit. Austen was hit or miss but overall I enjoyed it. It didn't damage the X-Men NEAR as much as Morrison's X-Men run. And despite complaints, I really enjoyed everthing Claremont's done with the X-Men save the later half of X-Treme. I loved the arc with the Fury. I liked the Neo. I LOVED X-Treme through the Invasion story. Excalibur was fantastic. I think he's had some good stuff. The second half of X-Treme wasn't even that bad, just mediocre with bad art that further hurt it.
Personally, I don't think the X-Men books started going downhill until Milligan and Brubaker were on the two books. The later half of Whedon's run was during that time but it was so plagued with delays there was little enjoyment in it, so even that was tainted at the time (though great in hindsight).