On one hand, I am playing without time traveling. On the other hand, I am trading with people online who are all-but-certainly time traveling.
I think it is entirely your call. I kind of like the slow burn and with time traveling, I'd probably be a bit more obsessive about getting the "perfect" set up with villagers and plots. Without time travel, I can sit back and enjoy the game as it is meant to be enjoyed.
I agree with Chet, I think it really depends on you, and what kind of environment you're wanting to play in!
I think if you're looking for a more relaxed time, probably minimal to no tt! Personally think some tt (like advancing a day) isn't too harmful, but probably nothing extreme (weeks, etc.) if it's meant to be casual.
Like the previous posters said, it's up to you. If the thought of trying a time travel town seems interesting to you, I'd say go for it.
I personally only play utilizing time travel because I've done so since the GameCube game and I enjoy it way more than playing on a daily basis. It makes the game less of a chore to be beholden to and more of a fun thing you can do at your leisure. No stress over flowers wilting or villagers moving out unexpectedly if you don't play for a few days. You can play multiple days in a single day if you don't want to wait for stores to open or renovate or whatever else. I don't see how others would find it somehow less relaxing, I find it more relaxing.
For information's sake, if you play any other games that might punish you for messing with the 3DS clock like Pok?mon locking you out of certain daily events for 24 hours, and even if you don't, I recommend changing the time with Isabelle instead of changing the 3DS internal clock itself. Also, even if you don't time travel all the time, it can be a good idea to keep track of the last day and time you played after each play session so if you ever take a long break you can travel back to that same day and not lose any villagers or suffer any environmental setbacks like flower wilting or weed growth. If you do that, the game figures it's still that exact same day and it will be like nothing happened even if you didn't touch the game for a year or longer. Heck, if you bought something that day at one of the stores then it would still be sold out.
Overall, yeah, if it interests you then give it a shot.