I've noticed the placement of villager houses, outdoor items + fences, and impassible features like cliffs and rivers can make it seem this way. Villagers will often take the easiest route to the plaza and back, at least that's what they seem to do in my town. Those whose houses are near each other's will spend time outside each other's houses just 'cause that's where they end up after wandering aimlessly around. 
So maybe check if there is a cliff or river blocking them off from going where they would ordinarily go. Their AI isn't good enough to try and find a way around it if there isn't an easily accessible bridge or something, so they might just be chilling out in the locations which they can access most easily.
So maybe check if there is a cliff or river blocking them off from going where they would ordinarily go. Their AI isn't good enough to try and find a way around it if there isn't an easily accessible bridge or something, so they might just be chilling out in the locations which they can access most easily.