I TT when I want to get ugly villagers out of my town and sometimes just for fun when I'm bored I don't see anything bad in it, I've never lost any of my villagers accidentally while TTing even I've been doing it a lot.
I used to love time-traveling until I realized that I was getting bored with the game. I've played every Animal Crossing installment in the series and I've always time-traveled, so I recently gave waiting a shot and now I love it. It makes picking up the game in the morning that much more thrilling. <3