It takes some time, I mostly just dont talk to them and ignore them if they try to approach me. Once they're ready to move they'll have a cloud over their head and you have to talk to them and convince them to move away.
Once they agree to move they'll be in boxes for an extra day ^-^