They only know who you are if its the exact same villager who left your town, so if you had somebody holding them for you while you did the cycle. As far as I'm aware, and from what I've experienced, they are able to remember the last town they lived in.
If you do the cycle without having somebody to hold your villager, once it is completed and you get your villager back they won't make any reference to you or living in your town before when you come to collect them. They will just say they've ever seen you before.
When ed moved from my town he went into a friends, he recognised me just fine when I went to visit. But only because he was my ed from my town.
- - - Post Merge - - -
Oh, apologies, just realised I probably still didn't answer your question. After the 16 villager cycle, the villager, wherever you get them from, will still only remember the town they last lived in. So if somebody is holding your villager and you visited them, they will remember you up until you've cycled 16 villagers out. Then, they will remember their last town
Not very good at explaining things and it's early
