Luckily nothing bad has ever happen in my town when having visitors over. I do tons of trades, but they're over quickly, and people who want to hangout will have a good WIFI feedback rating on here. I'd get suss if anyone wandered off during a trade, as that's just bad manners, but I've been fortunate enough to only meet cool people so far ^_^
Twice.
The first time was when I added someone on another forum, and they wanted to visit my town. Since I've been to theirs before, it seemed fair. But then he got out his axe and chopped down one of my Perfect Orange trees. I told him to stop after the first chop, too. And I kinda took some of his hybrids the next time that I went to his town.
Second case was when I added this Japanese player on Club Tortimer. He gave me free Bells, and wanted me to add him, so I did. Afterwards, I left and a few hours later, I needed to open up my gates to trade with another people on TBT (for bush starts, if I recall correctly...). Just before I was about to wrap everything up, my game tells me that 'someone was coming'. I thought "Okay???", and saw it was him. I said 'Hi', but he just ran straight out of my Train Station and started to take my Black Tulips that were put around my house. I didn't know about flipping the Wi-Fi switch off at that time, so I just hurriedly pressed Save & Quit. Luckily only one black tulip was taken, but then I deleted his Friend Code after that.
When I first got ACNL, I added people from the island all the time. They took up a majority of my friend list, and they were all nice enough. I even made some really good friends that way, later on. For some reason, I really have no idea why, I thought it'd be fun to leave my gate open with my 3DS plugged in while I was in another room entirely for a few hours. I guess I thought my friends could come over if they felt like it, look around, take fruit, whatever.
Keep in mind that at the time, I had VERY specific signs everywhere. I had tiles that said "please walk" and "don't take flowers". I also laid out lots of presents for visitors. I made it clear what you could take and what you couldn't, and even had tiles that said "take free" when it was something that was a present.
So I go away for a few hours and, bet you didn't see this one coming (sarcasm), 4 black roses are gone. I guess it wasn't a big deal, but I really didn't think that it would happen to me.
In conclusion, although the island is a great way to meet friends, and sometimes you can find some really great people, I take care not to trust anyone too much.
I've had a few bad experiences, with the same person. She comes into my town only to either steal my things or trap me and the person I'm trading with in my town. I always flip the switch, so it's all good.