i would definitely invite whoever you are going to invite to your now open villager plot, or let a random villager move in properly - like all 3-4 days of the move-in process (the previous villager in boxes move out day, the open plot day/plot sold if you invite someone to your town, the random villager plot sold day, and the final move in house built day) BEFORE you time travel backwards!
TT backwards in any way during the move-in process has been known to glitch villager houses. this is a non destructive glitch - it doesn't harm anything but it does replace the moving in villager's house exterior with the previous' villagers'. if this happens to be something you want, feel free - but the safest way is to move forward day by day, entering the game, saving and exiting, to ensure no glitch occurs, for every single day of the move in process. once they move in fine, then you can TT backwards without worry.
& yeah, the game treats ANY amount of TT backwards as if you moved forwards one single day, so no need to worry about anything happening in excess there.
good luck!