The returned haven't aged, so schools will be inundated with new students the system hasn't had to account for. Seriously ill or injured patients will most likely die before receiving proper treatment. (You're no longer on the waiting list for a transplant, you'd have to be re-entered in a care program, etc.) Never mind the people who died as a direct result of the snap, not just those who were dusted. i.e., the people in aircraft whose pilots were dusted, people in surgery whose surgeons were dusted, etc.
What if you come back and your family has died of natural causes while you were away? The justice system is screwed. How many dusted people were awaiting trial or victims of a crime which never went to trial? You'd suddenly have billions of unemployed people who've missed out on five years of technological advancement, facing a global economic system not prepared to feed twice as many mouths with the same resources it had yesterday.
All this because Tony didn't want to sacrifice his daughter, who still would've been born anyway. Kind of a dick move when you think about it.