I do think it's a nice feature with a cool idea behind it, but I can see it becoming very frustrating if you're looking for a certain "aesthetic" for your town. Thankfully, it's never bothered me too much and I haven't really experienced it in my towns.