That's kinda an unfair question, because as much as I like this game, there are MANY things I would like to change, and-
... oh wait, so can I actually rant about it to my heart's content? That's nice!
Joking aside, I'm telling the truth. The losing villagers you mentioned is one of them, of course, but it's not the worst by far, FAR. How about getting a say in where the new neighbours place their houses? And I know this has been complained upon greatly by most of the players.
Same as diving creatures going past boundaries where you can't follow

But diving controls themselves also need improvement, in my opinion.
Aaand... the simplest of my complains, but one that annoys me immensely, is about furniture. There are some pieces that look like you can put things on, but you CAN'T! Like some dressers, wardrobes, tables, libraries, etc. Several of my decorating plannings got ruined because of that, I thought I could put a clock or a lamp on a dresser that would look very nice... and I found out it wasn't possible!

SO FRUSTRATING!!