I do, because I feel like the game would be way too difficult without it. It'd be very difficult to take a break from my town because of villagers randomly moving and my town would turn to a mess. Plus villagers would start to forget who I am and my friendship with them would decrease. I don't really have enough patience to wait for certain events or special characters in town, though I time travel for plot resetting and such as well.
