the downsides to time-travelling are: if you don't have beautiful town ordinance, then you may get extra weeds and dead flowers; your turnips will spoil; and you risk a villager moving out if you time travel for too many days in a row. (one way to protect against move-outs though: if you deny a villager move-out request, you are then safe to time travel and nobody will have moved out in the interim)
- - - Post Merge - - -
also, when you time travel backwards, no matter how far back you go, the game counts it as only one day for measuring things like villager move outs etc...