It feels brutal to say it, but it's the fact that some villagers look better and cuter than others. When the only difference between villagers of the same personality type is their design, why would I want to have villagers with an un-appealing design over villagers whose design I do like?
I mean, it is kind of sad, but it's not surprising to me that the deciding factor is often looks. Even when we pick villagers according to a theme, we do it based on looks. Maybe in this case we don't simply pick the ones who look nicest to us but the ones who fit the theme while also preferably looking nice.
For me it's mostly character design and sometimes having formed an attachment to a certain villager. When I started Solstice, I was planning on swapping out Tom for Lobo ASAP, but he was the first villager in that new town that formed a friendship with my mayor so I kept him for a while. Now I've had him there for years and I couldn't imagine the town without him. He's my OG villager