Yeah, they kind of suck that way. The best way I've found to get a villager out quick is the greet and then ignore method. I've been able to get my most recent move-in out this way only having to deny one other villager's request to move before he packed up.