Whenever I get their photo I let them leave and I try ones I haven't seen yet Its getting harder because I've seen nearly all the villagers and there are some that I didn't haven't see yet.
Same here I have a lot of amiibo cards with villagers who I've seen already and had before. I also have a Nook Shopping Catalog of Villager Posters (Which you can get from Harvs Island) that lets me know who I had before so that way I will know who to skip when I see them on the Nook Mile island. If I don't have them then I invite them.I do this as well, unless it’s a villager who I have an amiibo card for and they ask. I’ll let them go and then invite them invite them back down the road.
I also have a “priority” list of villagers to ask to move-in if I see them while island hunting or if they come to camp, ie, ones I don’t have an amiibo card for. But that list has dwindled, so it takes longer to find them.