Have you layed the spots for where villagers can put their homes like this?
OXXXO
OXXXO
OXXXO
OOXOO
Also, you should not cover your whole town with tiles, as it would simply negate the paths and act as if there aren't any, leading to a random spot.
X:Open Spot
O:Covered Tile