If your town has 10 villagers, and you visit another town you won't get any voided villagers from that town. This is because your town is full, therefore you can't take any more villagers. I suggest when visiting a town make sure the person you're visiting has a clear void. This "void" is where a villager that they recently had, but didn't go to a new home or didn't get picked up goes to. They can clear this void by visiting a town that has 10 villagers, if not the villager will remain in the void up until 30 days. When others are visiting your town they can also pick up your void, so you'd also have to go to a town with 10 villagers to avoid others getting your void. So my best advice is to make sure your town is full at all times! or if not, just don't have anyone over for a while until that 10th space is filled.