From my Time Travelling experience I never had any trouble with TT'ing backwards. BUT depending on what your reason is for TT'ing I think the safest time to travel backwards is after someone moves out / someone moves in / or when you have 8 villagers.
As has been said, TTing backwards any amount of time only counts as one day forward. It won't have any effect on your villagers, except if one of them is planning to leave tomorrow. Then they'll be in boxes.