Well, even Rebels basically said there's precedence for this. After Ezra and the Ghost crew kicked the Empire off of Lothal, they never came back. Based on Filoni's comments at the finale premiere, he gave the impression the Ghost's stand scared him and he was focused on Luke rather than retaking Lothal. So basically, Lothal remained free of Imperial oppression and presence from that time through the rest of the civil war. But Lothal was a more major planet with a lot better resources. With that in mind, it could totally work that there were even some backwater planets that were just so under the radar the Empire just didn't pay attention to them. Or they were so barren or unassuming, no one cared about them other than the people who were already there.