As soon as 16 other villagers have moved out after that villager, you can invite that specific villager to move back in. If it's the same one you "lost", they'll remember you and be happy to be neighbors again.

If it's a different one, they'll treat you as if they're meeting you for the first time.
If you're talking about getting that villager to show up again via random move in, it's kind of up for debate. I heard somewhere that you have to have all 333 villagers in the game before you can start randomly encountering villagers you've let move before, but I'm not sure if this is true or not.