After shaking a perfect tree a few certain times, the perfect fruit will not regrow anymore and will lose leaves and will yield 2 perfect fruits and rotten perfect fruit.
I don't think your fruit will rot. The only time rotten fruit should appear is in the scenario Crossing123 explained above. I've never taken a break, but I did time travel by months at a time in a spare weeding town once and my fruit never rotted.