The main problem is that you could lose a villager. I am always talking about how TTing made me lose Flurry!
(but at least she went to a lovely home with Patypus, and now I have Marina from Moonflow, & am just as happy with my little pink Octopus.)
but I lost her with just 2 days of time travel!
I still TT, but ONE day at a time & rarely (e.g. I was looking forward to Freya's birthday for ages, then I was busy that day & had to TT backwards to give her my gift)
but that was just 1 day.
So I do TT every now and then---but only by one day.
As for TTing by hours---I do that all the time when the shop is shut. I TT a few hours on a regular basis & it does nothing that I have noticed.
In other words TT a few hours here and there, or even a day...but be very careful & plan if you intend to TT more than 1 day....you could lose a villager.
If you plan to TT more than 1 day, ask for help before you do it & people will tell you what to check for.