Marvel comics has an underlying explanation as to why Earth seems to be at the centre of everything: the biological life has capacity for mutation, innovation and evolution that doesn't exist on other more advanced planets, and therefore merits their interest; in the MCU, this is alluded to by things like the Inhumans, the need for Asgardians to protect Earth as one of its "realms," the visitation to Earth by one Celestial and likely even more, etc. So just by being a high-traffic area it makes it likely that the stones have passed Earth's way at some point. And our inherent need to possess and exploit resources we don't quite understand means that should a stone pass our way, it's going to be kept here (i.e., the space, mind and time stones).