I don't really feel bad when I pass up villagers on mystery islands. I mean, they're just vacationing, so it's not like you're leaving them to die when you don't invite them to live with you. But what I do feel bad about is failing villager quests. The other day Piper asked me to catch her a stump insect, so I went and caught a violin beetle. Then I guess my brain just registered the quest as complete even though I didn't actually give her the bug yet, and so I went to bed and then woke up and saw the bug in my inventory, which is when I realized my mistake. I felt so bad, I did the apologetic reaction to her even though she's programmed to forget that she even asked for a bug in the first place.
Another thing that happened, is a while ago I had an open slot in my town, but before I could find one of my dreamies on a mystery island it got taken up by my friend's voided Rooney. So I was a little bit upset, and decided to use the method of just ignoring him and only talking to him when he has a thought bubble that could mean he wants to move out. But then this morning one of my villagers mentioned that he got sick, and I guess something just switched for me. I mean, I still don't intend for him to be a permanent resident of my island (I don't need a third cranky) and when he wants to move out I'll let him, but that doesn't mean I should just shun him! So, I got him his medicine and for however long he stays here he's gonna get treated like any of my other villagers.