Also, there are occasionally people who would attempt to steal/destroy towns, which of course, makes people reluctant.
Finally, because of the "void" people would occasionally not want any villagers, as sometimes it can mess up a layout. So many wait to play with others once their town is complete.