Well based off of Thor 1, I'd assume his flight was really all him and not the hammer, because he hovered in the tornado without twirling it, and regardless of the wind, if he needed the hammer he would've needed it to stabilize himself in the whirlwind
But in TDW, they made it pretty clear he cannot fly without the hammer, when he jumped off the balcony and the hammer came and carried him away, and also when he takes the tube to get back to the battle at the end while hammer-less
Avengers made little indication either way