It seems to be rather... up in the air right now. There's a lot of conflicting information. I watched a video that stated that hitting them with nets and the like actually makes them stay, since you want to utterly ignore them to get them to leave. However, I've had villagers that I definitely want to keep try to move out even though I talk to them regularly, so the "befriend them to get them to leave" seems to check out as well.
Anyway, no idea if this is accurate, but this is the video I watched:
http://youtu.be/6jIuEm8Bq9I