Wait until you have a villager you want to move trying to leave. Then you know who will leave by not playing and as only 1 can leave at a time you know the others are safe.
I've heard that after someone asks you to move, no one else can move until the next time you start up the game. So before you quit get one of your villagers to ask to move. You can tell them no, then no one else can move until the next time you fire up the game.