You can leave 2 tiles empty. Any 3x3 space will be okay for villagers to move into, so make sure you don't have any of those on the spots you don't want villagers to move in!
For example, this is where a villager WOULD NOT be able to move in (X=pattern, O=empty):
OXOOXOOX
OOOOOOOO
OOOOOOOO
OXOOXOOX