I don't get it, even with someone moving and then moving in you still have 8 villagers, which doesn't actually tell anyone if you even had 6 like the op said...
I think maybe you just miscounted, and/or didn't notice someone had a house plot since it doesn't show up on a map? Because you don't ever get 6 villagers unless you restart. The game is programmed so when you are down to 8, someone will move in again, so then someone can move out, but it definitely doesn't just go to 6..