I'm told that if you pick up the same instance of that villager, like if you sent them to another town, cycled 16, and then invited that villager back from that town, they'll recognize you and suggest that it'd be great if you were neighbors again. I don't know if it's true.
If it's a different copy of that villager, they definitely won't recognize you, though.