You can still go to your villages' house to hang out with them. The thing is you cannot walk freely in your town because there is actually no map in this game. The purpose of the map is to choose an environment for the villages to build their house, so multiple villages can live in the same area...
In the game play, the street only have one floor in the beginning and we can see the downstairs is a forest. Later, there is a stair that goes to the 2nd floor. Is it possible that there is a 3rd floor?
I think this time the town is actually this street, the place where the villagers live is...