Just like to add it doesn't even have to be recent nor does it have to be going to your town... The villager I got that crushed my garden early on in the game was from a friend visiting and it was a villager that had moved from their town weeks earlier--not just that, but they moved 2 people since. And I had never even been to her town so...
In general, if you have a space and don't want anyone moving in, you should turn StreetPass off/your wireless switch off and not let anyone move in or out until you have that spot taken or no longer care. Or have things positioned like that one Dream Village where nobody can move in.
Or quickly move someone in as it should overwrite the random person moving in and then you have control over it if you have less than 4 characters.