When a villager moves out of your town, they will enter "The Void." If you visit someone else's town, the villager in your Void will move into their town.
To avoid having a random villager move into a town of someone you are trading with, you need to clear your void.
To clear your void, you must visit a person who has 10 villagers, so no villager can move in.