I think its just because maybe there is an open space to build a house, maybe when plot resetting try using different characters responses/genders to see if they move. But if not, then they just want to move a bad spot like always - . -
^ THAT
(and they all insist on planting themselves literally right in front of your house, despite there are 3 million acres of available land around. Repeatedly.) -__- Plot resetting is tedious work.
I'm in the same situation. In the beginning of my 2nd town, a lot of the houses were settled near each other and I really liked that because it made it seem more towny. I don't want the houses to be scattered all over the place because in my 1st town, it's like that, and I don't really like that.