Well, it would pretty much be exactly as if they were voided ^^ When a villager doesn't go to a specific town, you can assume they go to some nonexistent location (another town, if you will) and basically they occasionally visit yours. I can't explain why it happens either way, other than that Nintendo wanted us to have a way to see our old villagers again!