Personally I have 8 permanent villagers, and I keep two slots free to meet new faces. I like having my permies, they are mostly villagers I met randomly/in dream towns/saw them up for adoption, and then grew to love and couldn't imagine them leaving. I've never lost any because I'm really careful if I have to TT, I always make sure to take my time and not rush, and I keep talking to every villager so I don't miss a ping. If I ever had to take a break from playing, I'd make a note of the last date played, and then start again on that day and slowly TT forward to catch up. I think if you take a few precautions, you won't have to worry too much about losing dream villagers