I only wanted 1 visitor to move out, and I ignored him and occasionally wacked hi mwi ha net on the head. I don't know if that sped things up or not though.
but I've seen people recommend actually increasing friendship with the villager you don't like, on the theory that the game increases the odds of your higher level friendship villagers moving out... (don't know whether it's true though)